Warning! Executing cBots downloaded from this section may result in loss of funds. Use them at your own risk.
Notification Publishing copyrighted material is strictly prohibited. If you believe there is copyrighted material in this section you may use the Copyright Infringement Notification form to submit a claim.

General info

joeatbayes's avatar
Name:joe Ellsworth
Member since: 06 Dec 2014
Country:United States
Favorite symbols:USDCHF
CTO & Chief Algorithms Scientist Bayes Analytic LLC. Specializing in Machine Learning and Artificial Intelligence applied to predicting future prices.
We are expensive but we also solve problems others found impossible or too difficult. See: http://BayesAnalytic.com

Last comments

  • @Dump trendbars data to CSV: See: http://ctdn.com/algos/cbots/show/588  and http://ctdn.com/algos/cbots/show/591
  • @Data Export Ticks - Saves to CSV: Hey Deansi,   I took the time to add the order book values you mentioned.   Let me know if it was what you intended. 
  • @Data Export Ticks - Saves to CSV: I found a free data source with older data http://ratedata.gaincapital.com  it takes a little work to coerce the file formats to match.  I have found that working with tick data it doesn't seem to do much good to feed more than 200 days into our Machine Learning algorithms but the longer time frames are helpful when back-testing to see how the engine accommodates changing market conditions. 
  • @Data Export Ticks - Saves to CSV: Hey King,   Good point on the localized numbers.    I think it would be better to use \t as the delimiter so excel can still load it as delimited file than ";"   Densi,  Thanks for the tip on the MarketDepth.  I will change that as it makes more sense to feed into external engines.       The isBacktesting flag is much appreciated. I updated my bar downloader to attempt to backfill correctly but don't have time to do the same with the Tickdata. Perhaps in Jan.     


DescriptionSubmited byDateCategoryPreviewDownloadsCommentsRating

No uploaded.

DescriptionSubmited byDateCategoryPreviewDownloadsCommentsRating
Exports chosen bars for Symbol at chosen at the chosen bar size from calgo to CSV file in Directory specified.      Files can be imported by excel but do not open in  when trying to update as excel takes an exclusive lock which will crash the process. OK: Help Needed:   When I run this in Backtest mode with 1 minute bars I get something that looks like valid volumes.  When I run it with 10 tick bars I get the same volume in every bar.    How do we get valid volume for the smaller duration bars.  Question:  How do you convert (MarketSeries.TickVolume to actual Transaction volume for the Bar. I want Total number of units traded during the bar.   I think transaction volume is a more meaningful input for momentum because a small transaction of a fractional unit doesn't mean as much as if a 100 million $ changed hands during the same change in price.  Recommended used is to use the Backtester to download as much historical data as possible in a back-test then switch over to activating the strategy live.  It will auto-backfill the data between end of Backtest and live data but use the Backtest to get as much of the data as possible first as it's lastBar semantic is simplistic.  I have found that tickvolume is only accurate when the Backtest is using the "Tick Data from Server"       The system is smart enough to add new data from current Backtest to end of existing data and it will not overwrite or duplicate the older data but it is not smart enough to detect gaps if the user runs a Backtest covering a couple of months then skips a few months and runs a different Backtest. When used live it will write a new bar whenever a bar has been completed for as long as the cbot is active.    The files are opened in shared mode so a separate process can read new data from them as they arrive.  I needed this because our AI prediction engine needs new data as it arrives and can not run inside of calgo.  I will eventually  write the reverse plumbing to allow predictions from the external engine to be picked up by calgo and executed as trades. When used in back-tester will export the data in the time-frame specified by the backtest.   I tested it back to Jan-1-2011 and all the data seemed to show up.         File name is derived from the symbol, bar duration and chosen moving average and looks like:  exp-EURUSD-Minute-ma3-bars.b1.csv Data  Looks as follows: datetime,open,close,high,low,volume,weighted_val,rsi,stddev,spread 2014-01-27 00:00:00.000,1.368700,1.368630,1.368730,1.368610,23.000000,1.368650,67.318063,0.000223,0.000050 2014-01-27 00:01:00.000,1.368620,1.368590,1.368630,1.368590,15.000000,1.368600,66.005144,0.000282,0.000050 2014-01-27 00:02:00.000,1.368570,1.368860,1.368880,1.368490,41.000000,1.368773,70.226266,0.000365,0.000060 2014-01-27 00:03:00.000,1.368880,1.368980,1.369010,1.368880,29.000000,1.368963,71.896500,0.000438,0.000050 2014-01-27 00:04:00.000,1.368970,1.368980,1.368980,1.368970,4.000000,1.368978,71.896500,0.000483,0.000030 2014-01-27 00:05:00.000,1.369010,1.368810,1.369010,1.368810,20.000000,1.368860,65.829147,0.000504,0.000070 2014-01-27 00:06:00.000,1.368800,1.368610,1.368810,1.368610,28.000000,1.368660,59.470578,0.000500,0.000080 2014-01-27 00:07:00.000,1.368590,1.368970,1.368980,1.368550,36.000000,1.368868,65.862483,0.000511,0.000060 2014-01-27 00:08:00.000,1.368980,1.368990,1.369000,1.368980,6.000000,1.368990,66.181582,0.000512,0.000060 In Excel it looks like: Note: To auto parse the dates when loading in excel change the code to replace T between dateTime with a space. Enhanced Dec-21-2014: To only add bars newer than what are already present in the file but it is still missing the code  to detect gaps and try to fill them in.   Enhanced Dec-212-2014 - To attempt to backfill bars between current bar and the last bar it finds in the file.  This is needed because the the back tester in CAglo always runs a little behind the current data and would create a gap of some very critical missing current data.     This process calculates how many are missing and attempts to backfile from current series.   I have tested it over a weekend but not over longer periods of times.     Want to collaborate:   www.linkedin.com/pub/joe-ellsworth/0/22/682/  or http://BayesAnalytic.com No Promises,  No Warranty.  
22 Dec 2014
Runs as a bot  that writes tick data to a  external CSV file  that can be read by other processes.    I needed this because our AI algorithm exists outside of calgo and needs fresh data every tick to make it's predictions.     I will eventually build the plumbing to allow external buy / sell recommendations to be read back from another stream using a separate bot. For those who want bar data instead of tick data.  Look for DataExportBars. Enhanced 2014-12-21 - Added the num_bid, num_ask and vol_adj_ask, vol_adj_bid columns based on market depth also removed some columns that have values that never changed.   I think  vol_adj_bid and vol_adj_ask show where the market really is with less gaming.  I like this because it shows where the bulk of the real orders are concentrated.   That way the odd sell at a very low or very high price people use to try and game the system are averaged out because they typically do not use large volumes for this kind of gaming.   It may be even better to throw away the statistical outliers and average the rest. Enhanced:2014-12-21 - Reverse spread for vol_adj_price and added a price adjust volume to provide a basic bull vs bear indicator based on where the bulk the open order book is hanging.   When used against backtest it can download historical data.  (See instructions in source) but the Market depth values I use to set num_bid, num_ask, vol_adj_ask, vol_adj_bid are not available during back test which is a real shame because I think these would be really handle for Machine learning algorithms.   The output file name is constructed from the Symbol plus the length of the moving average.  An example is: exp-EURUSD-ma30-ticks.csv   The moving average is included as an additional field just to show how it can be merged with the tick data. Sample Output: date,ask,bid,spread,num_ask,num_bid,vol_adj_ask,vol_adj_bid,vol_adj_spread 2014-12-22 06:51:29.746,1.225520,1.225410,0.000110,11,10,1.225600,1.225347,-0.000252 2014-12-22 06:51:36.527,1.225520,1.225420,0.000100,9,8,1.225590,1.225356,-0.000234 2014-12-22 06:51:44.308,1.225520,1.225430,0.000090,8,10,1.225584,1.225355,-0.000229 2014-12-22 06:51:44.621,1.225520,1.225440,0.000080,7,11,1.225588,1.225357,-0.000231 2014-12-22 06:51:47.621,1.225540,1.225440,0.000100,9,10,1.225594,1.225358,-0.000235 2014-12-22 06:51:49.371,1.225540,1.225460,0.000080,12,9,1.225635,1.225345,-0.000290 2014-12-22 06:51:49.621,1.225550,1.225460,0.000090,12,13,1.225662,1.225332,-0.000330 2014-12-22 06:51:50.371,1.225560,1.225460,0.000100,12,11,1.225675,1.225336,-0.000339 2014-12-22 06:51:50.605,1.225570,1.225460,0.000110,11,13,1.225682,1.225334,-0.000348 2014-12-22 06:51:50.918,1.225550,1.225450,0.000100,11,12,1.225684,1.225337,-0.000348 2014-12-22 06:51:51.168,1.225570,1.225490,0.000080,11,15,1.225667,1.225319,-0.000348 2014-12-22 06:51:51.480,1.225600,1.225520,0.000080,13,12,1.225723,1.225392,-0.000331 2014-12-22 06:51:51.621,1.225620,1.225520,0.000100,11,11,1.225703,1.225413,-0.000290 2014-12-22 06:51:52.418,1.225610,1.225520,0.000090,9,12,1.225676,1.225436,-0.000241 Sample in Excel When ran live can save new inbound ticks to the file as they arrive where a external process can pick them up.   I opened the file in shared mode so both processes can access the file simultaneously. It delivers a poor mans pipe for freshly arriving bar data.    I tested this in Lua and it will sit there waiting for new bars to show up at the end of the file and seems to catch them every time they arrive.     When using in this mode I recommend using a fast SSD or Ramdisk to minimize IO latency to disk. I apologize to those who want to use it internationally since I know the comma delimiter will mess up your formatting.  I tried changing it to a Tab delimited format but then it takes 5 extra clicks to load into excel.  There are only two lines where you have to change the , to tab to make  it work internationally.  I did not try to fix the back fill data problem for ticks like I did with bars because my application did not find them essential.  Let me know if you want this feature we can not duplicate it exactly but should be able to get close except for being able to backfill the gaps between end of back-test and real time.  Note:  To avoid saving ticks that had no order book I added a If Statement if ((volAdjAsk == 0.0) && (volAdjBid == 0.0)) it will have to be commented if you want to use it to download historical bars.  Want to collaborate:     www.linkedin.com/pub/joe-ellsworth/0/22/682/ or     http://bayesanalytic.com No Promises,  No Warranty.  Terms of use MIT http://opensource.org/licenses/MIT
23 Dec 2014