indicator 목록
https://www.quantconnect.com/docs/algorithm-reference/indicators#Indicators-Reference-Table
.
.
fine filter로 사용가능한 900정도의 지표 목록
https://www.quantconnect.com/docs/data-library/fundamentals#Fundamentals-Reference-Tables
if not self.indicator.IsReady:
returnschedule jobs
# schedule an event to fire at a specific date/time
self.Schedule.On(self.DateRules.On(2013, 10, 7),
self.TimeRules.At(13, 0),
self.SpecificTime)
# schedule an event to fire every trading day for a security the
# time rule here tells it to fire 10 minutes after SPY's market open
self.Schedule.On(self.DateRules.EveryDay("SPY"),
self.TimeRules.AfterMarketOpen(self.spy, 10),
self.EveryDayAfterMarketOpen)
# schedule an event to fire every trading day for a security the
# time rule here tells it to fire 10 minutes before SPY's market close
self.Schedule.On(self.DateRules.EveryDay("SPY"),
self.TimeRules.BeforeMarketClose("SPY", 10),
self.EveryDayBeforeMarketClose)
# schedule an event to fire on certain days of the week
self.Schedule.On(self.DateRules.Every(DayOfWeek.Monday, DayOfWeek.Friday),
self.TimeRules.At(12, 0),
self.EveryMonFriAtNoon)
# the scheduling methods return the ScheduledEvent object which can be used
# for other things here I set the event up to check the portfolio value every
# 10 minutes, and liquidate if we have too many losses
self.Schedule.On(self.DateRules.EveryDay(),
self.TimeRules.Every(timedelta(minutes=10)),
self.LiquidateUnrealizedLosses)
# schedule an event to fire at the beginning of the month, the symbol is
# optional.
# if specified, it will fire the first trading day for that symbol of the month,
# if not specified it will fire on the first day of the month
self.Schedule.On(self.DateRules.MonthStart("SPY"),
self.TimeRules.AfterMarketOpen("SPY"),
self.RebalancingCode)
.
indicators
hese are provided in two ways: through helper short cut methods, and as class objects. Indicators created through the short cut methods have already been wired up to receive data and are “ready to use”
.
One key indicator to learn is the Identity indicator, which simply returns the value of the asset.
pep = Identity("PEP") # Pepsi ticker
coke = Identity("KO") # Coke ticker
delta = IndicatorExtensions.Minus(pep, coke) # Difference between them
.
Indicator Ready
if not self.indicator.IsReady: return
.
Algorithm Warm-Up
def Initialize(self):
self.AddEquity("SPY", Resolution.Hour)
# define a 10-period daily RSI indicator with shortcut helper method
self.rsi = self.RSI("SPY", 10, MovingAverageType.Simple, Resolution.Daily)
# set a warm-up period to initialize the indicator
self.SetWarmUp(timedelta(20))
# Warm-up the indicator with bar count
# self.SetWarmUp(10, Resolution.Daily)
Universe Selection does not support warm-up and, consequently, factors that depend on indicators are not updated.
.
History Request Warm-Up
def Initialize(self):
self.AddEquity("SPY", Resolution.Hour)
# define a 10-period daily RSI indicator with shortcut helper method
self.rsi = self.RSI("SPY", 10, MovingAverageType.Simple, Resolution.Daily)
# initialize the indicator with the daily history close price
history = self.History(["SPY"], 10, Resolution.Daily)
for time, row in history.loc["SPY"].iterrows():
self.rsi.Update(time, row["close"])
.
Basic Indicator Usage
The indicator resolution can be different from the resolution of your securities data. However, the resolution of the indicator should be equal to or higher than the resolution of your security.
def Initialize(self):
# request the hourly equity data
self.AddEquity("SPY", Resolution.Hour)
# define a 10-period daily RSI indicator with shortcut helper method
self.rsi = self.RSI("SPY", 10, MovingAverageType.Simple, Resolution.Daily)
def OnData(self, data):
# check if this algorithm is still warming up
if self.rsi.IsReady:
# get the current RSI value
rsi_value = self.rsi.Current.Value
# get the current average gain of rsi
average_gain = self.rsi.AverageGain.Current.Value
# get the current average loss of rsi
average_loss = self.rsi.AverageLoss.Current.Value
.
# request the daily equity data
self.AddEquity("SPY", Resolution.Daily)
# define a 10-period RSI indicator with indicator constructor
self.rsi = RelativeStrengthIndex(10, MovingAverageType.Simple)
# register the daily data of "SPY" to automatically update the indicator
self.RegisterIndicator("SPY", self.rsi, Resolution.Daily)
.
RegisterIndicator()를 통하면 indicator에 필요한 값들이 자동 업데이트 된다.
# request the equity data in minute resolution
self.AddEquity("SPY", Resolution.Minute)
# define a 10-period RSI indicator with indicator constructor
self.rsi = RelativeStrengthIndex(10, MovingAverageType.Simple)
# create the 30-minutes data consolidator
thirtyMinuteConsolidator = TradeBarConsolidator(timedelta(minutes=30))
self.SubscriptionManager.AddConsolidator("SPY", thirtyMinuteConsolidator)
# register the 30-minute consolidated bar data to automatically update the indicator
self.RegisterIndicator("SPY", self.rsi, thirtyMinuteConsolidator)
.
Manual Update
def Initialize(self):
self.AddEquity("SPY", Resolution.Daily)
self.rsi = RelativeStrengthIndex(10, MovingAverageType.Simple)
def OnData(self, data):
# update the indicator value with the new input close price every day
if data.Bars.ContainsKey("SPY"):
self.rsi.Update(data["SPY"].EndTime, data["SPY"].Close)
# check if the indicator is ready
if self.rsi.IsReady:
# get the current RSI value
rsi_value = self.rsi.Current.Value
.
Updating Indicators with Custom Values
# define a 10-period daily RSI indicator with shortcut helper method
# select the Open price to update the indicator
self.rsi = self.RSI("SPY", 10, MovingAverageType.Simple, Resolution.Daily, Field.Open)
# define a 10-period RSI with indicator constructor
self.rsi = RelativeStrengthIndex(10, MovingAverageType.Simple)
# register the daily High price data to automatically update the indicator
self.RegisterIndicator("SPY", self.rsi, Resolution.Daily, Field.High)
.
Indicator Extensions
.
Plotting Indicators
self.Plot(string chart, Indicator[] indicators)
#Plot array of indicator objects; extending "Indicator" type.
self.Plot("Indicators", sma, rsi);
#Plot array of indicator objects; extending "TradeBarIndicator" type.
self.Plot("Indicators", atr, aroon);
#Currently, there is a limit of 4 indicators for each Plot call
#For complex plotting it might be easiest to simply plot your indicators individually.
.
Rolling Window
RollingWindow is an array of data that allows for reverse list access semantics, where the object with index [0] refers to the most recent item in the window, and index [Length-1] refers to the last item in the window,
self.closeWindow = RollingWindow[float](4) self.tradeBarWindow = RollingWindow[TradeBar](2) self.quoteBarWindow = RollingWindow[QuoteBar](2)
self.closeWindow.Add(data["SPY"].Close) self.tradeBarWindow.Add(data["SPY"]) self.quoteBarWindow.Add(data["EURUSD"])
currentClose = self.closeWindow[0] previousClose = self.closeWindow[1] oldestClose = self.closeWindow[self.closeWindow.Count-1]
# In Initialize, create the rolling windows
def Initialize(self):
# Create a Rolling Window to keep the 4 decimal
self.closeWindow = RollingWindow[float](4)
# Create a Rolling Window to keep the 2 TradeBar
self.tradeBarWindow = RollingWindow[TradeBar](2)
# Create a Rolling Window to keep the 2 QuoteBar
self.quoteBarWindow = RollingWindow[QuoteBar](2)
# In OnData, update the rolling windows
def OnData(self, data):
if data.ContainsKey("SPY"):
# Add SPY bar close in the rolling window
self.closeWindow.Add(data["SPY"].Close)
# Add SPY TradeBar in rolling window
self.tradeBarWindow.Add(data["SPY"])
if data.ContainsKey("EURUSD"):
# Add EURUSD QuoteBar in rolling window
self.quoteBarWindow.Add(data["EURUSD"])
.
Combining with Indicators
# In Initialize, create the rolling windows
def Initialize(self):
# Creates an indicator and adds to a rolling window when it is updated
self.SMA("SPY", 5).Updated += self.SmaUpdated
self.smaWindow = RollingWindow[IndicatorDataPoint](5)
# Adds updated values to rolling window
def SmaUpdated(self, sender, updated):
self.smaWindow.Add(updated)
Indicators emit an Updated event after they have been updated. To create a rolling window of indicator points, we attach an event handler function to Updated, which adds the last value of the indicator to the rolling window. The value is an IndicatorDataPoint object that represents a piece of data at a specific time.
.
Universe
Universe selection is the process of screening or filtering the assets you’d like to trade by some predetermined formula.
QuantConnect provides two universes; Coarse Universe, and Fine Universe for the US Equities Market. The QuantConnect Universe data is refreshed every day. You can also create custom universes of data which we’ll explore later in this section.
After passing through your filters, the algorithm needs you to return an array of Symbol objects. LEAN automatically subscribes to these new symbols and adds them to your algorithm.
When you deselect an asset you currently own or have open orders, it will continue receiving data. This is to ensure the portfolio models are accurate. You can use the algorithm self.ActiveSecurities property to get the list of assets currently in a universe. Additionally, even once removed from the universe the security object in the Securties collection is maintained for record keeping purposes (e.g. total fees accrued, volume traded).
By default, assets selected by universe selection are requested with minute resolution data. You can change the default selection by adjusting the UniverseSettings property, which we’ll dive into below.
.
Coarse Universe Selection
To use a coarse universe, you must request it using an AddUniverse() call from the Initialize() method of your algorithm. You should pass in a function that will be used to filter the stocks down to the assets you are interested in using.
class MyCoarseUniverseAlgorithm(QCAlgorithm): def Initialize(self): self.AddUniverse(self.MyCoarseFilterFunction) def MyCoarseFilterFunction(self, coarse): pass
def MyCoarseFilterFunction(self, coarse): sortedByDollarVolume = sorted(coarse, key=lambda x: x.DollarVolume, reverse=True) filtered = [ x.Symbol for x in sortedByDollarVolume if x.Price > 10 and x.DollarVolume > 10000000 ] return filtered[:500]
# setup state storage in initialize method
self.stateData = { }
def MyCoarseFilterFunction(self, coarse):
# We are going to use a dictionary to refer the object that will keep the moving averages
for c in coarse:
if c.Symbol not in self.stateData:
self.stateData[c.Symbol] = SelectionData(c.Symbol, 200)
# Updates the SymbolData object with current EOD price
avg = self.stateData[c.Symbol]
avg.update(c.EndTime, c.AdjustedPrice, c.DollarVolume)
# Filter the values of the dict to those above EMA and more than $1B vol.
values = [x for x in self.stateData.values() if x.is_above_ema and x.volume > 1000000000]
# sort by the largest in volume.
values.sort(key=lambda x: x.volume, reverse=True)
# we need to return only the symbol objects
return [ x.symbol for x in values[:10] ]
class SelectionData(object): def __init__(self, symbol, period): self.symbol = symbol self.ema = ExponentialMovingAverage(period) self.is_above_ema = False self.volume = 0 def update(self, time, price, volume): self.volume = volume if self.ema.Update(time, price): self.is_above_ema = price > ema
.
Take 10 stocks the furthest above their 10 day SMA of volume.
class SelectionData(object): def __init__(self, symbol, period): self.symbol = symbol self.volume = 0 self.volume_ratio = 0 self.sma = SimpleMovingAverage(period) def update(self, time, price, volume): self.volume = volume if self.sma.Update(time, volume): # get ratio of this volume bar vs previous 10 before it. self.volume_ratio = volume / self.sma.Current.Value
def CoarseFilterFunction(self, coarse): for c in coarse: if c.Symbol not in self.stateData: self.stateData[c.Symbol] = SelectionData(c.Symbol, 10) avg = self.stateData[c.Symbol] avg.update(c.EndTime, c.AdjustedPrice, c.DollarVolume) # filter the values of selectionData(sd) above SMA values = [sd for sd in self.stateData.values() if sd.volume > sd.sma.Current.Value and sd.volume_ratio > 0] # sort sd by the largest % jump in volume. values.sort(key=lambda sd: sd.volume_ratio, reverse=True) # return the top 10 symbol objects return [ sd.symbol for sd in values[:10] ]
.
Fundamentals Selection
Tip:
Only 5,000 assets have fundamental data. When working with fundamental data, you should always include the “HasFundamentalData” filter in your Coarse Universe filter.
.
Requesting a Fundamental Universe
To request a fundamental universe, pass a second filter-function into the
AddUniverse()
method.
class MyUniverseAlgorithm(QCAlgorithm): def Initialize(self): self.AddUniverse(self.MyCoarseFilterFunction, self.MyFineFundamentalFunction) def MyCoarseFilterFunction(self, coarse): pass def MyFineFundamentalFunction(self, fine): pass
.
From the top 50 stocks with the highest volume, take 10 with lowest PE-ratio.
# In Initialize: self.AddUniverse(self.CoarseSelectionFunction, self.FineSelectionFunction) def CoarseSelectionFunction(self, coarse): sortedByDollarVolume = sorted(coarse, key=lambda x: x.DollarVolume, reverse=True) filtered = [ x.Symbol for x in sortedByDollarVolume if x.HasFundamentalData ] return filtered[:50] def FineSelectionFunction(self, fine): sortedByPeRatio = sorted(fine, key=lambda x: x.ValuationRatios.PERatio, reverse=False) return [ x.Symbol for x in sortedByPeRatio[:10] ]
.
900 fundamental properties
https://www.quantconnect.com/docs/data-library/fundamentals#Fundamentals-Morningstar-US-Equity-Data
.
Universe Settings
Below is the UniverseSettings object and its default settings:
//Popular universe settings: self.UniverseSettings.Resolution // What resolution should added assets use .Leverage // What leverage should assets use in the universe? .FillForward // Should asset data fill forward? .MinimumTimeInUniverse // Minimum time assets should be in universe .ExtendedMarketHours // Should assets also feed extended market hours?
.
These settings should be configured in your Initialize() method before you request your universe. They are globals, so they will apply to all universes you create.
class MyCustomSettingsUniverseAlgorithm(QCAlgorithm): def Initialize(self): # Request second resolution data. This will be slow! self.UniverseSettings.Resolution = Resolution.Second self.AddUniverse(self.MySecondResolutionCoarseFilterFunction)
.
Configuring Universe Securities
When adding securities from the universe, some algorithms need raw or partially adjusted price data. This can be controlled by the SetSecurityInitializer() method.
#In Initialize self.SetSecurityInitializer(self.CustomSecurityInitializer) def CustomSecurityInitializer(self, security): '''Initialize the security with raw prices''' security.SetDataNormalizationMode(DataNormalizationMode.Raw)
축약형
# Most common request; requesting raw prices for universe securities. self.SetSecurityInitializer(lambda x: x.SetDataNormalizationMode(DataNormalizationMode.Raw))
.
Security Changed Events
When universe contents are changed (securities are added or removed from the algorithm), we generate an OnSecuritiesChanged event. This allows your algorithm to know the changes in the universe state. The event passes in the SecurityChanges object containing references to the Added and Removed securities.
def OnSecuritiesChanged(self, changes):
self._changes = changes
self.Log(f"OnSecuritiesChanged({self.UtcTime}):: {changes}")
a list of current securities is provided by the ActiveSecurities property. This is a dictionary of Security objects which are currently in your universe.
.
Universe Creation Short Cuts 축약형으로 universe 만들기
/ Helper: Add US-equity universe for the top 50 stocks by dollar volume self.AddUniverse(self.Universe.DollarVolume.Top(50)) // Helper: Add US-equity universe for the bottom 50 stocks by dollar volume self.AddUniverse(self.Universe.DollarVolume.Bottom(50)) // Helper: Add US-equity universe for the 90th dollar volume percentile self.AddUniverse(self.Universe.DollarVolume.Percentile(90)) // Helper: Add US-equity universe for stocks between the 70th and 80th dollar volume percentile self.AddUniverse(self.Universe.DollarVolume.Percentile(70, 80))
.
Custom Universe Selection
# add the custom universe in initialize self.AddUniverse(NyseTopGainers, "myCustomUniverse", Resolution.Daily, self.nyseTopGainers) # filter function using your custom data def nyseTopGainers(self, data): return [ x.Symbol for x in data if x["Rank"] > 5 ]
# Example custom universe data; it is virtually identical to other custom data types. class NyseTopGainers(PythonData): def GetSource(self, config, date, isLiveMode): return SubscriptionDataSource(@"your-remote-universe-data", SubscriptionTransportMedium.RemoteFile) def Reader(self, config, line, date, isLiveMode): # Generate required data, then return an instance of your class. nyse = NyseTopGainers() nyse.Time = date # define end time as exactly 1 day after Time nyse.EndTime = nyse.Time + timedelta(1) nyse.Symbol = Symbol.Create(symbolString, SecurityType.Equity, Market.USA) nyse["Rank"] = rank return nyse
.
History
All python history requests return a Data Frame, which has different columns depending on the type of data requested. Data is returned as TradeBars, QuoteBars, or Slice objects depending on how you request it and the data available for your security.
Return Format
- History requests with symbols provided return data as a pandas data frame.
- History requests without symbols provided fetch history for the entire universe of your securities, and return it as an array of
Sliceobjects.
Time Index
- History results are indexed by the EndTime of a bar.
History data is returned in an ascending order from oldest to newest. This is required for piping the data into indicators to perform warm-up.
.
Single Symbol Request Examples
# EXAMPLE 1: Requesting By Bar Count: 5 IBM TradeBars, defaulting to security resolution:
self.AddEquity(“IBM”, Resolution.Daily)
self.df = self.History(self.Symbol(“IBM”), 5)
.
Multiple Symbol Request Examples
self.df = self.History([self.Symbol("IBM"), self.Symbol("AAPL")], 2)
.
- Resolution: LEAN attempts to guess the resolution you request by looking at any securities you already have in your algorithm. If you have a matching Symbol, QuantConnect will use the same resolution. When no default values can be located
Resolution.Minuteis selected.
.
All Securities History Request
The parameters are very similar to other history methods, but the return type is an array of Slice objects. This has the same properties as the OnData() Slice object.
The Slice object holds all of the results in a sorted enumerable collection you can iterate over with a foreach loop.
# EXAMPLE 1: Requesting 5 Bars For All Securities, default to security resolution:
# Setting Up Universe
self.AddEquity("IBM", Resolution.Daily)
self.AddEquity("AAPL", Resolution.Daily)
# Request history data and enumerate results:
slices = self.History(5)
for s in slices:
print(str(s.Time) +
" AAPL:" + str(s.Bars["AAPL"].Close) + " IBM:" + str(s.Bars["IBM"].Close))
.
# EXAMPLE 2: Requesting 5 Minutes For All Securities: slices = self.History(timedelta(minutes=5), Resolution.Minute) for s in slices: print(str(s.Time) + " AAPL:" + str(s.Bars["AAPL"].Close) + " IBM:" + str(s.Bars["IBM"].Close))
.
Working with Data Frames
# Setup Universe:
eurusd = self.AddForex("EURUSD", Resolution.Daily).Symbol
nzdusd = self.AddForex("NZDUSD", Resolution.Daily).Symbol
# STEP 1: Request Dataframe:
self.df = self.History([eurusd, nzdusd], 3)
# STEP 2: Check if empty and lock onto a symbol index with the loc[] method: if not self.df.empty: eurusd_quotebars = self.df.loc["EURUSD"]
# STEP 3: Extract and manipulate a single column with the string column name: spread = eurusd_quotebars["askclose"] - eurusd_quotebars["bidclose"]
# STEP 3: Extract and manipulate a single column with the string column name: spread = eurusd_quotebars["askclose"] - eurusd_quotebars["bidclose"]
.
# UNSTACKING: Transform into columns:
# Fetch multi-indexed history:
self.dataframe = self.History([self.Symbol("IBM"), self.Symbol("AAPL")], 3)
# Transform using unstack: self.dataframe["close"].unstack(level=0)
.
Streaming Warm Up Period
# Wind time back 7 days from start: self.SetWarmup(timedelta(7)) # Feed in 100 bars before start date: self.SetWarmup(100)
# In Initialize
self.emaFast = self.EMA("IBM", 50);
self.emaSlow = self.EMA("IBM", 100);
self.SetWarmup(100);
// In OnData: Don't run if we're warming up our indicators.
if self.IsWarmingUp: return
.
Charting
self.Plot("Series Name", value)
# In Initialize(self)
self.macd = MACD（"SPY", 12, 26,9, MovingAverageType.Exponential, Resolution.Daily）
# In OnData(self, data)
self.Plot("My Indicators", "MACD Signal", self.macd.Signal)
.
Manually Creating Charts (좀더 자세하게 차트 customize하는 방법)
In your initialize method, you can use the AddChart(Chart obj) method to insert a new chart. Each chart object has an internal collection of Series objects.
In creating Series objects, you must specify the name of the series, the SeriesType , and the index the series operates on. The series index refers to its position in the chart – for example, if all the series are index 0, they will lay on top of each other. If each series has its own index, it will have several mini-charts stacked next to each other.
# In your initialize method:
# Note - use single quotation marks: ' instead of double "
# Chart - Master Container for the Chart:
stockPlot = Chart('Trade Plot')
# On the Trade Plotter Chart we want 3 series: trades and price:
stockPlot.AddSeries(Series('Buy', SeriesType.Scatter, 0))
stockPlot.AddSeries(Series('Sell', SeriesType.Scatter, 0))
stockPlot.AddSeries(Series('Price', SeriesType.Line, 0))
# Or using custom chart
# Import the necessary module before using Custom color
from System.Drawing import Color
stockPlot.AddSeries(Series('Price', SeriesType.Line, '$', Color.Green))
stockPlot.AddSeries(Series('Buy', SeriesType.Scatter, '$', Color.Red, ScatterMarkerSymbol.Triangle))
stockPlot.AddSeries(Series('Sell', SeriesType.Scatter, '$', Color.Blue, ScatterMarkerSymbol.TriangleDown))
self.AddChart(stockPlot)
# Later in your OnData(self, data):
self.Plot('Trade Plot', 'Price', data.Bars["SPY"].Close)
.
Supported Series Types
SeriesType.Line .Scatter .Candle .Bar .Flag
.
Custom Colors and Scatter Symbols
You can customize the chart color and the scatter symbol when creating the series. In your customized chart, instead of specifying the index, you need to specify the label name of the y-axis using a string like '$', '%', or an empty string ''. The chart color can be changed by specifying the parameter "Color.ColorName".
ScatterMarkerSymbol.Circle .Diamond .Square .Triangle .TriangleDown
.
Machine Learning
data storing
.
Logging and Debug
Algorithms can send debug messages to the console using the Debug(string message) method.
Algorithms can save more detailed messaging to log files for later analysis using Log(string message).
self.Debug(str(self.Time) + " Purchasing AAPL: " + str(slice["SPY"].Price))
self.Error("Volatility too high, terminating algorithm.")
self.Quit() # Optional: Instruct algorithm to stop.
self.Log("Additional detailed logging messages")
Reconcilation
QuantConnect annualizes the daily DTW. An annualized distance provides a user with a measurement of the annual difference in the magnitude of returns between the two curves. A perfect score is 0, meaning the returns for each day were precisely the same. A DTW score of 0 is nearly impossible to achieve, and we consider anything below 0.2 to be a decent score. A distance of 0.2 means the returns between an algorithm’s live and OOS performance deviated by 20% over a year.
What is Returns Correlation?
Returns correlation is the simple Pearson correlation between the live and OOS returns. Correlation gives us a rudimentary understanding of how the returns move together. Do they trend up and down at the same time? Do they deviate in direction or timing?
An algorithm’s returns correlation should be as close to 1 as possible. We consider a good score to be 0.8 or above, meaning that there is a strong positive correlation. This indicates that the returns move together most of the time and that for any given return you see from one of the curves, the other curve usually has a similar direction return (positive or negative).
.
.
QCAlgorithm, which provides some key helper properties for you to use: Security Manager, Portfolio Manager, Transactions Manager, Notification Manager, and Scheduling Manager. Along with hundreds of helper methods to make the API easy to use.
Securities property is a dictionary of Security objects. Each asset (equity, forex pair, etc) in your algorithm has a security object. All the models for a security live on these objects:
Portfolio is a dictionary of SecurityHolding classes. These classes track the individual portfolio items profit and losses, fees, and quantity held. e.g. self.Portfolio["IBM"].LastTradeProfit.
Other helpers like Transactions , Schedule , Notify , and Universe have their own helper methods,
def QCAlgorithm
{
self.Securities; # Array of Security objects.
self.Portfolio; # Array of SecurityHolding objects
self.Transactions; # Transactions helper
self.Schedule; # Scheduling helper
self.Notify; # Email, SMS helper
self.Universe; # Universe helper
# Set up Requested Data, Cash, Time Period.
def Initialize:
# Other Event Handlers:
def OnData(self, slice):
def OnEndOfDay(self, symbol):
def OnEndOfAlgorithm():
# Indicator Helpers
def SMA():
}
.
.
In backtests you can set your starting capital using the self.SetCash(float cash) method. In live trading this is ignored and your brokerage cash is used instead.
.
Backtesting uses the self.SetStartDate(int year, int month, int day) and self.SetEndDate(int year, int month, int day) methods
self.SetStartDate(2013,1,1) self.SetEndDate(2015,1,1) self.SetEndDate(datetime.now() - timedelta(1)) # Or use a relative date.
.
self.AddEquity("SPY") # Default to minute bars
self.AddForex("EURUSD", Resolution.Second) # Set second bars.
# Setting the data normalization mode for the MSFT security to raw (pay dividends as cash) self.Securities["SPY"].SetDataNormalizationMode(DataNormalizationMode.Raw);
.
.
warming up
Often algorithms need some historical data to prime technical indicators, or populate historical data arrays. Using the SetWarmUp(TimeSpan period) or SetWarmUp(int barCount) methods you can specify a warm-up period.
During the warm-up period you cannot place a trade.
Algorithms can use the bool IsWarmingUp property to determine if the warm-up period has completed.
self.SetWarmUp(200) # Warm up 200 bars for all subscribed data. self.SetWarmUp(timedelta(7)) # Warm up 7 days of data.
.
AccountType, MarginModel, SetBrokerageModel
In QuantConnect, we model your algorithm with margin-modeling by default, but you can select a cash account type. Cash accounts do not allow leveraged trading, whereas Margin accounts support 2-4x leverage on your account value.
Margin accounts with more than $25,000 in equity are eligible for pattern day trading margin limits. This increases your available leverage to 4x while the market is open and 2x overnight. To model this behavior in your algorithm, you must set your security MarginModel to PatternDayTradingMarginModel.
SetBrokerageModel(BrokerageName brokerage, AccountType account)
#Brokerage model and account type:
self.SetBrokerageModel(BrokerageName.InteractiveBrokersBrokerage, AccountType.Cash)
//Add securities and if required set custom margin models
spy = self.AddEquity("SPY") # Default to minute bars
spy.MarginModel = PatternDayTradingMarginModel()
.
Benchmark
self.SetBenchmark("SPY")
.
onData()
The primary event handler, Slice, groups all data types together at a single moment in time in the def OnData(self, slice): handler.
All data uses DataDictionary objects to group data by symbol and provide easy access to information. The plural of the type denotes the collection of objects (e.g., the TradeBars DataDictionary is made up of TradeBar objects; QuoteBars DataDictionary is made up of QuoteBar objects).
ibmTradeBar = slice.Bars["IBM"].
onData()를 통해서 slice obj가 전달된다. slice obj를 통해 각 ticker의 정보에 접근하는 방법은 아래와 같다.
- Dynamic string/symbol indexer, which returns a dynamic object of your type slice[“IBM”].
- Statically typed properties (slice.Bars[], slice.QuoteBars[]).
class Slice: TradeBars Bars; QuoteBars QuoteBars; Ticks Ticks; OptionChains OptionChains; FuturesChains FuturesChains; Splits Splits; Dividends Dividends; Delistings Delistings; SymbolChangedEvents SymbolChangedEvents;
it is possible to check if the time slice contains specific data. e.g. slice.ContainsKey("EURUSD") will return a boolean.
Even if requesting data to be filled forward, you should check if the data exists in the dictionary first.
.
Data Formats
There are seven financial data types: Tick, TradeBar, QuoteBar, Delisting, SymbolChangedEvent, Split, and Dividend. All data extends from BaseData, the core data class, which provides Symbol, Time, and Value properties.
.
TradeBars
self.AddEquity("IBM", Resolution.Hour) ## Subscribe to hourly TradeBars
def OnData(self, data):
## You can access the TradeBar dictionary in the slice object and then subset by symbol
## to get the TradeBar for IBM
tradeBars = data.Bars
ibmTradeBar = tradeBars['IBM']
ibmOpen = ibmTradeBar.Open ## Open price
ibmClose = ibmTradeBar.Close ## Close price
## Or you can access the IBM TradeBar by directly subsetting the slice object
## (since you are subscribed to IBM equity data, this will return a TradeBar rather than a QuoteBar)
ibmOpen = data['IBM'].Open ## Open price
imbClose = data['IBM'].Close ## Close price
.
Dividends
Dividend events are triggered on payment of a dividend. It provides the Distribution per share.
def Initialize(self):
self.SetStartDate(2017, 6, 1)
self.SetEndDate(2017, 6, 28)
self.spy = self.AddEquity("SPY", Resolution.Hour)
def OnData(self, data):
if not self.Portfolio.Invested:
self.Buy("SPY", 100)
## Condition to see if SPY is in the Dividend DataDictionary
if data.Dividends.ContainsKey("SPY"):
## Log the dividend distribution
self.Log(f"SPY paid a dividend of {data.Dividends['SPY'].Distribution}")
.
Splits
def Initialize(self):
self.SetStartDate(2003, 2, 1)
self.SetEndDate(2003, 2, 28)
self.SetCash(100000)
self.msft = self.AddEquity("MSFT", Resolution.Daily)
self.msft.SetDataNormalizationMode(DataNormalizationMode.Raw)
def OnData(self, data):
if not self.Portfolio.Invested:
self.Buy("MSFT", 100)
## If MSFT had a split, print out information about it
if data.Splits.ContainsKey("MSFT"):
## Log split information
spySplit = data.Splits['MSFT']
if spySplit.Type == 0:
self.Log('MSFT stock will split next trading day')
if spySplit.Type == 1:
self.Log("Split type: {0}, Split factor: {1}, Reference price: {2}".format(spySplit.Type, spySplit.SplitFactor, spySplit.ReferencePrice))
.
SymbolChangedEvent
SymbolChangedEvents provides notice of new ticker names for stocks, or mergers of two tickers into one. It provides the OldSymbol and NewSymbol tickers.
.
Delistings
Delisting events provide notice that an asset is no longer trading on the exchange. A delisting warning is issued on the final trading day for a stock delisting event to give your algorithm time to gracefully exit out of positions before forced termination.
.
using custom data
LEAN supports backtesting almost any external custom data source. To use this feature, you need to add the data during initialize using self.AddData() and instruct your algorithm on how to read your data. We provide helpers for popular data sources like Quandl and Intrinio, but if you are using your own format or server, you’ll need to create a custom type.
.
# In Initialize method: self.AddData(Weather, "KNYC", Resolution.Minute)
.
All data must extend from BaseData and override the Reader and GetSource methods.
.
class Weather(PythonData):
''' Weather based rebalancing'''
def GetSource(self, config, date, isLive):
source = "https://www.dropbox.com/s/8v6z949n25hyk9o/custom_weather_data.csv?dl=1"
return SubscriptionDataSource(source, SubscriptionTransportMedium.RemoteFile);
def Reader(self, config, line, date, isLive):
# If first character is not digit, pass
if not (line.strip() and line[0].isdigit()): return None
data = line.split(',')
weather = Weather()
weather.Symbol = config.Symbol
weather.Time = datetime.strptime(data[0], '%Y%m%d') + timedelta(hours=20) # Make sure we only get this data AFTER trading day - don't want forward bias.
weather.Value = decimal.Decimal(data[2])
weather["MaxC"] = float(data[1])
weather["MinC"] = float(data[3])
return weather
class MyAlgorithm(QCAlgorithm):
def Initialize(self):
self.SetStartDate(2019, 9, 20) # Set Start Date
self.SetCash(100000) # Set Strategy Cash
self.symbol = self.AddData(Weather, "KNYC", Resolution.Daily).Symbol
def OnData(self, data):
if data.ContainsKey(self.symbol):
maxC = data[self.symbol].GetProperty('MaxC')
minC = data[self.symbol].GetProperty('MaxC')
value = data[self.symbol].Value
.
Loading Reference Data
You may want to import a single static reference fileThis is possible with the Download() It can take header settings for authentication, and a username and password for basic authentication.
# If using dropbox remember to add the &dl=1 to trigger a download
csv = self.Download("https://www.dropbox.com?....&dl=1")
# read file (which needs to be a csv) to a pandas DataFrame. include following imports above
# from io import StringIO
# import pandas as pd
df = pd.read_csv(StringIO(csv))
.
financial data api
Intrinio Custom Data
# In Initialize method: self.AddData(IntrinioEconomicData, "$DCOILWTICO", Resolution.Daily)
https://docs.intrinio.com/documentation/web_api/get_all_economic_indices_v2
.
Tiingo Price Data
self.AddData(TiingoDailyData, "AAPL", Resolution.Daily)
.
Quandl Data
Quandl.SetAuthCode("my-quandl-api-token")
.
Consolidator
Consolidating data allows you to create bars of any length from smaller bars. Commonly this is used to combine one-minute price bars into longer bars such as 10-20 minute bars.
To achieve this, QuantConnect allows you to create Consolidator objects and register them for data.
self.Consolidate()
The consolidate helper accepts a timedelta, Resolution, or Calendar period specifier:
# Consolidate 1min SPY -> 45min Bars
self.Consolidate("SPY", timedelta(minutes=45), self.FortyFiveMinuteBarHandler)
# Consolidate 1min SPY -> 1-Hour Bars
self.Consolidate("SPY", Resolution.Hour, self.HourBarHandler)
# Consolidate 1min SPY -> 1-Week Bars
self.Consolidate("SPY", Calendar.Weekly, self.WeekBarHandler)
The event handler function of Consolidate accepts one argument, the resulting bar. For most data sources, this defaults to TradeBar format.
# Example event handler from Consolidate helper.
def FortyFiveMinuteBarHandler(self, consolidated):
self.Log(f"{consolidated.EndTime} >> FortyFiveMinuteBarHandler >> {consolidated.Close}")
.
Consolidating Data for Indicators
# Consolidating minute SPY into 14-bar daily indicators
ema = self.EMA("SPY", 14, Resolution.Daily)
sma = self.SMA("SPY", 14, Resolution.Daily)
# Generate 7 minute bars; then SMA-10 generates the average of last 10 bars.
self.AddEquity("SPY", Resolution.Minute)
self.sma = SimpleMovingAverage(10)
self.RegisterIndicator("SPY", self.sma, timedelta(minutes=7))
.
Rolling Window 와 Consolidated Bars를 조합한경우
# In initialize create a consolidator and add its bars to the window
self.window = RollingWindow[TradeBar](2)
self.Consolidate("SPY", timedelta(minutes=45), lambda x: self.window.Add(x))
# Now you can use the bar history; window[0] is current, window[1] is previous bar.
if self.window.IsReady and window[0].Close > window[1].Close:
self.Log("Current close price higher than the one 45 minutes ago")
.
Manually Consolidating Bar Count
def Initialize(self):
self.AddEquity("QQQ", Resolution.Hour)
threeCountConsolidator = TradeBarConsolidator(3)
threeCountConsolidator.DataConsolidated += self.ThreeBarHandler
self.SubscriptionManager.AddConsolidator("QQQ", threeCountConsolidator)
def ThreeBarHandler(self, sender, bar):
# With hourly data the bar period is 3-hours
self.Debug(str(bar.EndTime - bar.Time) + " " + bar.ToString())
def Initialize(self):
# Make sure you have the data you need
self.AddEquity("QQQ", Resolution.Minute)
# Create consolidator you need and attach event handler
thirtyMinuteConsolidator = TradeBarConsolidator(timedelta(minutes=30))
thirtyMinuteConsolidator.DataConsolidated += self.ThirtyMinuteHandler
# Register consolidator to get automatically updated with minute data
self.SubscriptionManager.AddConsolidator("QQQ", thirtyMinuteConsolidator)
def ThirtyMinuteHandler(self, sender, bar):
# Bar period is now 30 min from the consolidator above.
self.Debug(str(bar.EndTime - bar.Time) + " " + bar.ToString())
.
Data Consolidation Event Handler
# self.Consolidate() Event Handler def FortyFiveMinuteBarHandler(self, consolidated): pass # Manually Created Event Handler def ThirtyMinuteBarHandler(self, sender, consolidated): pass
.
Removing a Consolidator
효율성을 위해 제거하는게 좋다.
# Remove a consolidator instance from subscription manager algorithm.SubscriptionManager.RemoveConsolidator(self.symbol, self.myConsolidator)
.
Securities and portfolio
Algorithms have a Securities property that stores a Security object for each asset in your algorithm. Security objects hold the models (backtesting behaviors) and properties of an asset. Each security can be completely customized to behave as you’d like. Securities is a Dictionary<Symbol, Security> so you can access your Security objects with their ticker Securities["IBM"].Price.
// Popular Securities Property Values: self.Securities["IBM"].HasData # Security has data .Invested # Have holdings .LocalTime # Time on the asset exchange .Holdings # Portfolio object .Exchange # Exchange information .FeeModel; # Fee model setter
The Portfolio property is a collection of SecurityHolding objects to provide easy access to the holding properties. Portfolio also provides information about the whole portfolio state:
// Popular Portfolio Property Values: self.Portfolio.Invested # Hold at least one stock .Cash # Sum of all currencies in account (only settled cash) .UnsettledCash # Sum of all currencies in account (only unsettled cash) .TotalFees # Fees incurred since backtest start .TotalHoldingsValue # Absolute sum portfolio items .MarginRemaining # Remaining margin on the account .TotalMarginUsed # Sum of margin used across all securities .TotalPortfolioValue # Portfolio equity .TotalProfit # Sum of all gross profit .TotalUnrealizedProfit # Holdings profit/loss
The Portfolio class is a Dictionary<Symbol, SecurityHolding>, so it can be accessed via ticker index: Portfolio["IBM"].IsLong
// Popular Portfolio Property Values: self.Portfolio["IBM"].Invested .IsLong # IsLong, IsShort Holdings. .Quantity # Shares held. .UnrealizedProfit; # Holdings profit/loss .TotalFees # Fees incurred since backtest start .Price; # Asset price
Check out the Security (Securities objects), SecurityPortfolioManager class, and SecurityHolding (Portfolio objects) classes.
#Access to Security Objects with Securities: self.Securities["IBM"].Price
#Security object properties:
class Security {
Resolution Resolution;
bool HasData;
bool Invested;
DateTime LocalTime;
SecurityHolding Holdings;
SecurityExchange Exchange;
IFeeModel FeeModel;
IFillModel FillModel;
ISlippageModel SlippageModel;
ISecurityPortfolioModel PortfolioModel;
ISecurityMarginModel MarginModel;
ISettlementModel SettlementModel;
IVolatilityModel VolatilityModel;
ISecurityDataFilter DataFilter;
}
.
Active Securities
The ActiveSecurities lets you select the assets currently in your universe. This is useful for iterating over those securities from your universe selection. It has all the same properties as the Securities collection.
# Securities currently in the universe: self.ActiveSecurities["IBM"].HasData # Security has data .Invested # Have holdings .LocalTime # Time on the asset exchange .Holdings # Portfolio object .Exchange # Exchange information .FeeModel; # Fee model setter
.
Trading and orders
Manual orders can be fetched, updated, and canceled with Order Tickets. As orders are filled and updated, they generate events that notify your algorithm about their execution.
You can control how optimistic or pessimistic these fills are with transaction model classes. In live trading, this fill price is set by your brokerage when the order is filled.
.
Placing Orders
All of the methods return an OrderTicket, which you can use to update and cancel the orders. All orders should use the symbol object to place orders;
.
Updating Orders
To update an order, you must use its OrderTicket.
Orders are updated by passing a UpdateOrderFields object to the Update method. The Update method returns an OrderResponse to signal the success or failure of the update request.
# Tag an order on creation
ticket = self.LimitOrder("SPY", 100, 221.05, False, "New SPY trade")
# Tag order later
updateSettings = UpdateOrderFields()
updateSettings.LimitPrice = 222.00
updateSettings.Tag = "Limit Price Updated for SPY Trade"
response = ticket.Update(updateSettings)
# Validate the response is OK
if response.IsSuccessful:
self.Debug("Order updated successfully")
.
Cancel Orders
To cancel an order, you must use its OrderTicket
The Cancel() method returns an OrderResponse object to determine if the operation was successful.
# Create an order and save its ticket
ticket = self.LimitOrder("SPY", 100, 221.05, False, "SPY Trade to Cancel")
# Tag order later
response = ticket.Cancel("Canceled SPY Trade")
# Use order response object to read status
if response.IsSuccessful:
self.Debug("Order successfully canceled")
.
Order Slippage Impact
we highly recommend you include a slippage model in your algorithm. Slippage is the difference between the expected and final fill prices of a trade.
.
Transaction Cost Models
By default, transaction fees are modelled from Interactive Brokers Brokerage rates. These models are customizable by setting a new FeeModel.
.
Automatic Position Sizing (SetHoldings)
Single Asset Targets
This is a quick way to set up a portfolio with a set of weights for assets. If you already have holdings, you may want to liquidate the existing holdings first to free up buying power.
# Allocate 50% of buying power to IBM via market orders.
self.SetHoldings("IBM", 0.5)
# Allocate 50% of portfolio value to IBM, but liquidate other holdings before starting
self.SetHoldings("IBM", 0.5, True)
.
Portfolio of Asset Targets
The portfolio variant of SetHoldings was designed to do this operation for you by accepting an array of PortfolioTarget objects.
If you already have holdings, the LEAN engine will calculate the delta-order required to meet your new target. When required, positions will be scaled down before other positions are scaled up in size.
# Purchase a portfolio of targets, processing orders intelligently.
self.SetHoldings([PortfolioTarget("SPY", 0.8), PortfolioTarget("IBM", 0.2)])
.
Portfolio of Asset Targets
The portfolio variant of SetHoldings was designed to do this operation for you by accepting an array of PortfolioTarget objects.
If you already have holdings, the LEAN engine will calculate the delta-order required to meet your new target. When required, positions will be scaled down before other positions are scaled up in size.
# Purchase a portfolio of targets, processing orders intelligently.
self.SetHoldings([PortfolioTarget("SPY", 0.8), PortfolioTarget("IBM", 0.2)])
.
Manually Calculating Quantity Target
# Calculate the fee adjusted quantity of shares with given buying power
quantity = self.CalculateOrderQuantity("IBM", 0.4)
self.LimitOrder("IBM", quantity, self.Securities["IBM"].Price)
.
Cash Buffer Setting
buffer is set to 2.5%. The buffer lives on the algorithm Settings property.
# Adjust the cash buffer from the default 2.5% to 5% self.Settings.FreePortfolioValuePercentage = 0.05
.
Liquidating Portfolio
# Liquidate all IBM in your portfolio
self.Liquidate("IBM")
// Liquidate entire portfolio
self.Liquidate()
.
.
Managing Orders
The algorithm Transactions Manager (SecurityTransactionManager) is a collection of helper methods for quick access to all your orders. It is located in the self.Transactions property on your algorithm
Fetching Single Order
Using the Transactions Manager, you can retrieve a clone of an order by its Id. Once sent, orders cannot be changed, so the clone of the order is for informational purposes only. To update an order’s properties, you should use an Order Ticket.
# Retrieve a clone of a previously sent order. order = self.Transactions.GetOrderById(orderId)
.
Fetching All Open Orders
Using the Transaction Manager, you can fetch a list of all open orders for a symbol. This is helpful if you want to update multiple open orders for a specific symbol.
# Retrieve a list of all open orders for a symbol openOrders = self.Transactions.GetOpenOrders(symbol)
.
Canceling All Orders
The Cancel helpers can cancel all open orders, or just those orders related with a specific symbol. The method returns a list of OrderTicket objects.
# Cancel all open orders
allCancelledOrders = self.Transactions.CancelOpenOrders()
# Cancel orders related to IBM, apply string tag.
ibmCancelledOrders = self.Transactions.CancelOpenOrders("IBM", "Hit stop price")
.
Tracking Order Events
def OnOrderEvent(self, orderEvent):
order = self.Transactions.GetOrderById(orderEvent.OrderId)
if orderEvent.Status == OrderStatus.Filled:
self.Log("{0}: {1}: {2}".format(self.Time, order.Type, orderEvent))
status
.
Time In Force
# Set Limit Order to be good until market close
self.DefaultOrderProperties.TimeInForce = TimeInForce.Day
self.LimitOrder("IBM", 100, lastClose * decimal.Decimal(.999))
# Set Market Order to be good until noon
self.DefaultOrderProperties.TimeInForce = TimeInForce.GoodTilDate(datetime(2019, 6, 19, 12, 0, 0))
self.MarketOrder("IBM", 100)
.
By default, orders remain open until they are canceled (TimeInForce.GoodTilCanceled). To update the value, set the DefaultOrderProperties.TimeInForce before placing an order. Doing so will change the default value for all future orders unless reassigned again.
# Set Limit Order to be good until market close
self.DefaultOrderProperties.TimeInForce = TimeInForce.Day
self.LimitOrder("IBM", 100, lastClose * decimal.Decimal(.999))
# Set Market Order to be good until noon
self.DefaultOrderProperties.TimeInForce = TimeInForce.GoodTilDate(datetime(2019, 6, 19, 12, 0, 0))
self.MarketOrder("IBM", 100)
.
Market Orders
If you do not have sufficient capital for the purchase, your order will be rejected. By default, market orders are synchronous and fill immediately.
# Create a Market Order for 100 shares of IBM.
marketTicket = self.MarketOrder("IBM", 100)
self.Debug("Market Order Fill Price: {0}".format(marketTicket.AverageFillPrice))
.
Configuring Market Order Timeouts
# Adjust the market fill-timeout to 30 seconds. self.Transactions.MarketOrderFillTimeout = timedelta(seconds=30)
.
Asynchronously Sending Market Orders
# Create a Market Order for 100 shares of IBM asynchronously.
self.MarketOrder("IBM", 100, True)
.
Limit Orders
Limit orders can be updated
# Purchase 10 SPY shares when its 1% below the current price
close = self.Securities["SPY"].Close
limitTicket = self.LimitOrder("SPY", 10, close * .99)
.
Stop Market Orders
If the market gaps (jumps in a discontinuous manner) past your stop price, it may be filled at a substantially worse price than the stop price you entered. As such, a stop-loss order is no guarantee your trade will fill at the price you specify.
Stop Market Order StopPrice, Tag, and Quantity can be updated.
# Create Stop Market Order for 1% below current market price.
close = self.Securities["SPY"].Close
stopMarketTicket = self.StopMarketOrder("SPY", 10, close * 0.99)
.
Stop Limit Orders
close = self.Securities["SPY"].Close
stopPrice = close * .99 # Trigger stop limit when price falls 1%.
limitPrice = close * 1.01 # Sell equal or better than 1% > close.
stopLimitTicket = self.StopLimitOrder("SPY", 10, stopPrice, limitPrice)
.
Market On Open-Close Orders
The Market On Open Quantity and Tag properties can be updated after creation until the last two minutes before open.
The Market On Open Quantity and Tag properties can be updated after creation until the last two minutes before close.
# Create Market Open/Close Orders for 100 shares of IBM
marketOpenOrderTicket = self.MarketOnOpenOrder("SPY", 100) # Place Before Open
marketCloseOrderTicket = self.MarketOnCloseOrder("SPY", 100) # Place Before Close
.
Tagging Orders and Debugging
# Tag an order on creation
ticket = self.LimitOrder("SPY", 100, 221.05, "New SPY trade")
# Tag order later
updateSettings = UpdateOrderFields()
updateSettings.Tag = "Our New Tag for SPY Trade"
ticket.Update(updateSettings)
.
original source : https://youtu.be/e60ItwlZTKM
code:
https://github.com/joeyajames/Python/blob/master/Pandas/Pandas%20-%20Change%20Column%20Names.ipynb
.
.
pandas with datetime
code :
original source : https://github.com/joeyajames/Python/blob/master/Numpy/Python%20Numpy%20Intro.ipynb
Python Numpy Intro
An introduction to the Python Numpy numerical python library.
The core data structure behind Numpy is the n-dimensional Numpy Array. It is 3x to 10x faster and more memory efficient than Python’s lists because, similar to Java arrays, it uses contiguous blocks of memory, and all elements are the same data type so there is no type checking at runtime. The Numpy library also includes many built-in code-saving mathematical functions that can be performed on an entire array or any slice of an array with a single line of code (ie. no for loops).
Numpy n-dimensional arrays are also sometimes referred to as nd-arrays.
Install Numpy using pip: pip install numpy The convention for importing numpy is import numpy as np.
import numpy as np
Creating a Numpy Array
There are MANY ways to instantiate a numpy array. I covered the most common ones below. Docs here cover more constructors.
- Pass in a list to the array() constructor
- Use the arange function, similar to the range function but used for Numpy arrays. Uses arguments, (start, stop+1, step).
- Use linspace to create an array of n equally spaced values. Uses arguments (start, stop, number of items).
- Create an array empty, full of ones or zeros, or full of any fill value. Uses argument (shape) in the form of a tuple.
You can pass in dtype as an optional argument for any of these. This is especially useful if you want to limit memory usage for a very large array of small integers because int8 and int16 use much less space than the default int32.
In [96]:
a = np.array([1,3,5,7,9,11])
print(a)
a = np.arange(1, 12, 2) # (start, stop, step)
print(a)
a = np.linspace(5, 8, 13) # (start, stop, number of items)
print(a)
a = np.zeros((4, 2))
print(a)
a = np.ones((2, 3), dtype=np.int16)
print(a)
a = np.full((6,), 88)
print(a)
a = np.fromstring('25 30 35 40', dtype=np.int, sep=' ')
print(a)
a = np.array([[1,3,5],[7,9,11]])
print(a)
b = np.zeros_like(a) # _like gives you a new array in the same shape as the argument.
print(b)
[ 1 3 5 7 9 11] [ 1 3 5 7 9 11] [5. 5.25 5.5 5.75 6. 6.25 6.5 6.75 7. 7.25 7.5 7.75 8. ] [[0. 0.] [0. 0.] [0. 0.] [0. 0.]] [[1 1 1] [1 1 1]] [88 88 88 88 88 88] [25 30 35 40] [[ 1 3 5] [ 7 9 11]] [[0 0 0] [0 0 0]]
Numpy Array Attributes
Get size (number of items), shape (dimensions), itemsize(bytes of memory for each item), and dtype (numpy data type).
See how many bytes of memory space the whole array uses from the product of size and itemsize. See complete list of attributes and methods.
In [97]:
print(a.size) print(a.shape) print(a.ndim) print(a.itemsize) print(a.dtype) print(a.nbytes) # same as a.size * a.itemsize
6 (2, 3) 2 4 int32 24
Indexing and Slicing
Use square brackets to get any item of an array by index. Multi-dimensional arrays can use multiple square brackets.
There are three arguments for slicing arrays, all are optional: [start:stop:step].
If start is left blank it defaults to 0. If stop is left blank it defaults to the end of the array. Step defaults to 1.
In [98]:
print(a) print(a[1]) print(a[0][2]) print(b[2:4]) print(a[:1]) print(a[1:3:2]) print(a[:, 1:2]) # all elements on dimension 0, only element 1 on dimension 1
[[ 1 3 5] [ 7 9 11]] [ 7 9 11] 5 [] [[1 3 5]] [[ 7 9 11]] [[3] [9]]
Reshape, Swap Axes, Flatten
See full list of array manipulation routines.
In [99]:
c = np.arange(-9, -3,).reshape(2,3) print(c) c = c.swapaxes(0,1) print(c) c = c.flatten() print(c)
[[-9 -8 -7] [-6 -5 -4]] [[-9 -6] [-8 -5] [-7 -4]] [-9 -6 -8 -5 -7 -4]
Use dtype to Save Space
Default data types (int32 and float64) are memory hogs. If you don’t need the higher precision you can save a lot of memory space and improve speed of operations by using smaller data types. For large data sets this makes a big difference.
In [100]:
d = np.arange(0,100) print(d.dtype, type(d[1])) print(d.nbytes) d = np.arange(0,100, dtype='int8') print(d.dtype, type(d[1])) print(d.nbytes)
int32 <class 'numpy.int32'> 400 int8 <class 'numpy.int8'> 100
UpCasting, Rounding, Print Formatting
Data type of all Items is upcast to the most precise element.
In [101]:
e = np.array([(1.566666,2,3), (4,5,6)]) print(e.dtype) e = e.round(4) print(e) np.set_printoptions(precision=2, suppress=True) # show 2 decimal places, suppress scientific notation print(e)
float64 [[1.57 2. 3. ] [4. 5. 6. ]] [[1.57 2. 3. ] [4. 5. 6. ]]
Numpy Data Types Available
uint is unsigned int, for positive numbers.
In [102]:
import pprint as pp pp.pprint(np.sctypes)
{'complex': [<class 'numpy.complex64'>, <class 'numpy.complex128'>],
'float': [<class 'numpy.float16'>,
<class 'numpy.float32'>,
<class 'numpy.float64'>],
'int': [<class 'numpy.int8'>,
<class 'numpy.int16'>,
<class 'numpy.int32'>,
<class 'numpy.int32'>,
<class 'numpy.int64'>],
'others': [<class 'bool'>,
<class 'object'>,
<class 'bytes'>,
<class 'str'>,
<class 'numpy.void'>],
'uint': [<class 'numpy.uint8'>,
<class 'numpy.uint16'>,
<class 'numpy.uint32'>,
<class 'numpy.uint32'>,
<class 'numpy.uint64'>]}
Reading and Writing to Files
Can use loadtxt, or genfromtxt to load data to load an entire file into an array at once. Genfromtxt is more fault tolerant.
Use savetxt to write an array to file.
In [103]:
f = np.loadtxt('data.txt', skiprows=1, delimiter=',', dtype=np.int32)
print(f)
print(f.dtype)
np.savetxt('data2.txt', f, delimiter=';', fmt='%d', header='a;b;c;d;e;f;g;h;i;j', comments='')
[[9 3 8 7 6 1 0 4 2 5] [1 7 4 9 2 6 8 3 5 0] [4 8 3 9 5 7 2 6 0 1] [1 7 4 2 5 9 6 8 0 3] [0 7 5 2 8 6 3 4 1 9] [5 9 1 4 7 0 3 6 8 2]] int32
In [111]:
g = np.genfromtxt('data.txt', skip_header=1, delimiter=',', dtype=np.int32)
print(g)
[[9 3 8 7 6 1 0 4 2 5] [1 7 4 9 2 6 8 3 5 0] [4 8 3 9 5 7 2 6 0 1] [1 7 4 2 5 9 6 8 0 3] [0 7 5 2 8 6 3 4 1 9] [5 9 1 4 7 0 3 6 8 2]]
Mathematical Functions
Numpy has an extensive list of math and scientific functions.
The best part is that you don’t have to iterate. You can apply an operation to the entire array or a slice of an array at once.
In [105]:
print(g > 4) print(g ** 2 - 1)
[[ True False True True True False False False False True] [False True False True False True True False True False] [False True False True True True False True False False] [False True False False True True True True False False] [False True True False True True False False False True] [ True True False False True False False True True False]] [[80 8 63 48 35 0 -1 15 3 24] [ 0 48 15 80 3 35 63 8 24 -1] [15 63 8 80 24 48 3 35 -1 0] [ 0 48 15 3 24 80 35 63 -1 8] [-1 48 24 3 63 35 8 15 0 80] [24 80 0 15 48 -1 8 35 63 3]]
In [106]:
print(g.min()) print(g.max()) print(g.sum()) print(g.mean()) print(g.var()) # variance print(g.std()) # standard deviation print(g.sum(axis=1)) print(g.min(axis=0)) print(g.argmin()) # index of min element print(g.argmax()) # index of max element print(g.argsort()) # returns array of indices that would put the array in sorted order
0 9 270 4.5 8.25 2.8722813232690143 [45 45 45 45 45 45] [0 3 1 2 2 0 0 3 0 0] 6 0 [[6 5 8 1 7 9 4 3 2 0] [9 0 4 7 2 8 5 1 6 3] [8 9 6 2 0 4 7 5 1 3] [8 0 3 9 2 4 6 1 7 5] [0 8 3 6 7 2 5 1 4 9] [5 2 9 6 3 0 7 4 8 1]]
Column Operations
Apply functions only to specific columns by slicing, or create a new array from the columns you want, then work on them.
But Beware that creating a new pointer to the same data can screw up your data if you’re not careful.
In [113]:
print(g[:, 2:3]) print(g[:, 2:3].max()) col3 = g[:, 3:4] # not a copy, just a pointer to a slice of g print(col3.std()) col3 *= 100 # Beware: this is applied to g data print(g)
[[8] [4] [3] [4] [5] [1]] 8 298.607881119482 [[ 9 3 8 70000 6 1 0 4 2 5] [ 1 7 4 90000 2 6 8 3 5 0] [ 4 8 3 90000 5 7 2 6 0 1] [ 1 7 4 20000 5 9 6 8 0 3] [ 0 7 5 20000 8 6 3 4 1 9] [ 5 9 1 40000 7 0 3 6 8 2]]
Numpy Random Functions
In [ ]:
np.set_printoptions(precision=5, suppress=True) # show 5 decimal places, suppress scientific notation h = np.random.random(6) print(h) h = np.random.randint(10, 99, 8) # (low, high inclusive, size) print(h) np.random.shuffle(h) # in-place shuffle print(h) print(np.random.choice(h)) h.sort() # in-place sort print(h)
original source : https://ohgyun.com/416 by 꿀벌개발일지
협정 세계시(UTC)
협정 세계시(UTC)는 1972년 1월 1일부터 시행된 국제 표준시이다.
“UTC"는 보통 "Universal Time Code"나 "Universal Time Convention"의 약어로 사용되기도 하는데,이는 틀린 것이라 한다.실제로 국제 전기 통신 연합은 통일된 약자를 원했지만,영어권의 사람들과 프랑스어권의 사람들은 각각 자신의 언어로 된 약자를 사용하길 원했다.영어권은 CUT(Coordinated Universal Time)을, 프랑스어권은 TUC(Temps Universel Coordonne)를 제안했으며,결국 두 언어 모두 C, T, U로 구성되어 있는 것에 착안해 UTC로 약어를 결정하기로 했다.
UTC는 그리니치 평균시(GMT)로 불리기도 하는데,UTC와 GMT는 초의 소숫점 단위에서만 차이가 나기 때문에 일상에서는 혼용되어 사용한다.기술적인 표기에서는 UTC가 사용된다.
참고로 그리니치 평균시(Greenwich Mean Time, GMT)는,런던(영국의 수도)을 기점으로 하고 웰링턴(뉴질랜드의 수도)을 종점으로 하는 협정 세계시의 빠른 시간이다.
UTC는 그레고리력의 표기를 따른다.1일은 24시간으로 나뉘며, 1시간은 60분으로, 1분은 60초로 나뉘는 것이 보통이나 약간은 가변적이다.UTC의 하루는 보통 86,400초이다. (24시간)
그러나 실제 태양의 자전/공전으로 측정하는 태양시는 86,400초보다 약간 길기 때문에,UTC에서는 때때로 하루의 제일 마지막 1분을 61초로 계산하기도 한다.이렇게 추가되는 초를 윤초라고 하고, 주로 12월 31일이나 6월 30일의 마지막에 추가한다.1972년 이후 25회를 적용했으며,가장 최근에는 2012년 6월 30일 오후 11시 59분 59초(UTC)에서 2012년 7월 1일 0시(UTC)로 넘어갈 때 적용하였다.
그레고리력과 윤년
4로 나뉘어 떨어지는 해는 원칙적으로 윤년이며, 2월 29일을 갖는다.하지만, 100으로 나뉘어 떨어지는 해라면 평년으로 한다.만약, 400으로 나뉘어 떨어진다면 윤년을 유지한다.
ISO 8601
ISO 8601은 날짜와 시간의 표기에 관한 국제 표준 규격이다.정식 명칭은 아래와 같다.Date elements and interchange formats – Information interchange – Representation of dates and times최신 버전인 ISO 8601:2004와 구버전인 ISO 8601:2000, ISO 8601:1988 이 있다.
ISO 8601에 따라 날짜를 기록하는 방법에는 3가지가 있다. – 연월일로 기록하는 방법 – 연과 연중 일수로 기록하는 방법 – 연과 주와 주중 일수로 기록하는 방법이다.
날짜의 표기ISO 8601에는 날짜의 표기에 그레고리력을 따른다.
– 연을 표기할 때는 기원전 1년에 해당하는 0000부터 기원후 9999년에 해당하는 9999까지의 값을 사용할 수 있다. 단, 율리우스력이 사용되던 0000년부터 1582년까지의 값은 정보 교환 시 상호 합의 하에 그레고리력으로 환산해야 한다. 0000년 전이나 9999년 후의 표기도 역시 사용하는 주체간의 상호 합의가 필요하다. 예를 들어, 연도 앞에 빼기표(-)를 붙이면 기원전 1년 이전, 더하기표(+)를 붙이면 기원전 1년 이후를 가리키는 식으로 합의할 수 있다. – 월의 표기는 01부터 12까지의 값을 가지며, 각각 1월부터 12월까지를 나타낸다. – 일의 표기는 01부터 시작하며 달에 따라 28부터 31까지의 값을 가진다. – 연중 일의 표기는 001부터 시작하며 해에 따라 365에서 366까지의 값을 가진다. – 연중 주의 표기는 01부터 시작한다. – 주중 일의 표기는 월요일을 1로 시작하여 순서대로 요일에 숫자를 붙이며, 일요일을 7로 한다.
연월일 표기법 YYYY-MM-DD (확장 형식) 또는 YYYYMMDD (기본 형식)으로 표기한다. 예) 1981-02-22 또는 19810222 : 1981년 2월 22일
연과 연중 일수 표기법 YYYY-DDD (확장 형식) 또는 YYYYDDD (기본 형식)으로 표기한다. DDD는 연중 날의 번호로 1월 1일이 001이며, 12월 31일은 평년은 365, 윤년은 366이 된다. 예) 1981-053 또는 1981053 : 1981년의 53번 째 날
연과 주의 주중 일수 표기법 YYYY-Www-D (확장 형식) 또는 YYYYWwwD (기본 형식)으로 표기한다. ww는 연중 주의 번호로, 해의 첫 주는 01, 마지막 주는 52 또는 53이 된다. 단, 해의 첫 주는 그 해의 첫번째 목요일을 포함한 주로 한다. D는 요일을 나타내며, 월요일이 1, 일요일이 7이다. 예) 1981-W07-7 또는 1981W077 : 1981년의 7번째 주의 일요일
시간의 표기
시간의 표기에는 쌍점을 쓴 hh:mm:ss (확장 형식) 또는 hhmmss (기본 형식)을 사용한다.hh는 시를 나타내며 00부터 24까지의 값을 가지고,mm은 분으로 00부터 59까지의 값을, ss는 초로 00부터 59까지의 값을 갖는다.
날짜와 시간을 함께 표기할 때에는, 날짜와 시간 사이에 T를 넣어 표기한다.
예) 1981-02-22T09:00:00 : 1981년 2월 22일 09:00
시간대의 표기
시간대를 표기할 때에는 Z또는 +/- 기호를 사용한다.
UTC 시간대에서는 시각 뒤에 Z를 붙인다.예) 1981-02-22T09:00Z 또는 19810222T0900Z : UTC 시간대에서의 1981년 2월 22일 오전 9시
UTC 외의 시간대에서는 시각 뒤에 +- hh:mm, +- hhmm, +- hh 를 덧붙여 쓴다.예) 1981-02-22T09:00:00+09:00 : UTC+9 시간대에서의 1981년 2월 22일 오전 9시
+가 붙으면, UTC의 시각보다 더 "빠르다"다는 의미다. 반대로 -는 느리다는 것을 의미하다.예를 들어, 1981-02-22T09:00+09:00 는 1981-02-22T00:00Z 와 동일하다.즉, UTC+9 시간대에서는 오전 9시이지만, UTC 시간대에서는 오전 0시이다.
기간의 표기
기간을 나타낼 때에는 시작일시/종료일시로 표기한다.예) 1981-02-22/2007-09-26 1981-02-22T09:00:00+09:00/2007-09-26T17:00:00+09:00
RFC 3339 Date Formats
ISO8601을 인터넷 프로토콜로 어떻게 다룰 것인지를 규정한 RFC 이다.ISO8601과 거의 비슷하며, 약간의 차이만 있을 뿐이다.예를 들면, RFC 3339에서는 ’T’의 생략을 허용하지 않고, 날짜와 시간 사이의 공백을 허용한다.대부분의 경우, 이 둘을 상세하게 분리해서 생각하지 않아도 된다.
참고:- UTC 위키– 그레고리력 위키– ISO 8601 위키– 타임존 맵 위키– RFC 3339