joe Ellsworth
We are expensive but we also solve problems others found impossible or too difficult. See: http://BayesAnalytic.com
joe Ellsworth's avatar

Info

Username:joeatbayes
Name:joe Ellsworth
Member since: 06 Dec 2014
Country:United States

Favorite symbols:

USDCHF

About

CTO & Chief Algorithms Scientist Bayes Analytic LLC. Specializing in Machine Learning and Artificial Intelligence applied to predicting future prices.

Signature

We are expensive but we also solve problems others found impossible or too difficult. See: http://BayesAnalytic.com

Last Algorithm Comments

@Dump trendbars data to CSV:  26 Dec 2014, 00:29


See: /algos/cbots/show/588  and /algos/cbots/show/591

@Data Export Ticks - Saves to CSV:  22 Dec 2014, 08:56


Hey Deansi,   I took the time to add the order book values you mentioned.   Let me know if it was what you intended. 

@Data Export Ticks - Saves to CSV:  22 Dec 2014, 06:32


I found a free data source with older data http://ratedata.gaincapital.com  it takes a little work to coerce the file formats to match.  I have found that working with tick data it doesn't seem to do much good to feed more than 200 days into our Machine Learning algorithms but the longer time frames are helpful when back-testing to see how the engine accommodates changing market conditions. 

@Data Export Ticks - Saves to CSV:  22 Dec 2014, 06:27


Hey King,   Good point on the localized numbers.    I think it would be better to use \t as the delimiter so excel can still load it as delimited file than ";"   Densi,  Thanks for the tip on the MarketDepth.  I will change that as it makes more sense to feed into external engines.       The isBacktesting flag is much appreciated. I updated my bar downloader to attempt to backfill correctly but don't have time to do the same with the Tickdata. Perhaps in Jan.     

Last Forum Posts

@Limit Order Failing for TradeType.Sell but not TradeType.Buy "INVALID_REQUEST" Works fine in Backtest but not Live:  06 Apr 2015, 19:37


PlaceLimitOrder(TradeType.Sell, Symbol, Symbol.VolumeMin, 1.07478, "EMACrosX1", 24 37, Time.AddMinutes(0.25), "Bid=1.07406 Ask=1.07413 targetLimitPrice=1.07406 stopPips=24.00000 profitPips=37.00000 LimitPriceEMA=1.07406 VeryShortEMA=1.07417 ShortEMA=1.07423275951391 LongEMA=1.07425 stopLimitPriceAdj=0.002470338 atRiskPerShare=0.00254444814 MaxCapitalAtRiskForSingleTrade=9.8989 maxShares=3890.39172950092 totalMaxRisk=7.63334442 orderVolume=3000 stopPips=24 profitPips=37 Account.FreeMargin=989.89 Account.Equity=989.89 PipSize=0.0001 Symbol.PipValue=0.0001 workingLongStopLimitPort=0.0023");

@Limit Order Failing for TradeType.Sell but not TradeType.Buy "INVALID_REQUEST" Works fine in Backtest but not Live:  03 Apr 2015, 02:27


Taking the standard debugging approach of removing complexity until  it worked and then adding things back in until it stopped I found that the failure point occurred when I added the comment string parameter.   It worked fine with the stopPips and takeProfitPips which I previously thought was causing the issue.   I have not tried to figure out exactly what it was about my comment string the server had a problem with and there is no documentation I can find about the limits of what can be put in a comment string.   This documentation would be very helpful.

@Limit Order Failing for TradeType.Sell but not TradeType.Buy "INVALID_REQUEST" Works fine in Backtest but not Live:  31 Mar 2015, 23:45


March-31 13:37

I modified the code to set the stopPips and ProfitPips to null and the order executed as expected but since the stop limit  and target price are null the order to add missing stop limits had to kick in to modify the position to add them.   There is some extra latency when used this way that leaves the position exposed to greater risk than desired.    Next I will modify to try just setting the profitPips and see if it still works. 

I modified the code From (FAILS):

      var orderResult = PlaceLimitOrder(TradeType.Sell, Symbol, orderVolume, targetLimitPrice, StratLabel, stopPips, profitPips, expireTime, cmt);

TO: (WORKS)
      var orderResult = PlaceLimitOrder(TradeType.Sell, Symbol, orderVolume, targetLimitPrice, StratLabel, nullnull, expireTime, null);

From The Journal

31/03/2015 20:26:08.276 | Limit order to Sell 0.03 Lot(s) EURUSD (Price: 1.07478, ExpireDate: 31/03/2015 20:26) is sent to server

31/03/2015 20:26:08.495 | → Limit order OID5366660 to Sell 0.03 Lot(s) EURUSD (Price: 1.07478, ExpireDate: 31/03/2015 20:26) is ACCEPTED (31/03/2015 20:26:08.357 UTC+0)

31/03/2015 20:26:08.792 | → Limit order to SELL 0.03 Lot(s) EURUSD OID5366660 is FILLED at 1.07478, position PID3227196 (31/03/2015 20:26:08.645 UTC+0)

31/03/2015 20:26:08.839 | Request to amend position PID3227196 (SL: 1.07725, TP: 1.07231) is sent to server

31/03/2015 20:26:09.042 | → Request to amend position PID3227196 (SL: 1.07725, TP: 1.07231) is ACCEPTED

@Limit Order Failing for TradeType.Sell but not TradeType.Buy "INVALID_REQUEST" Works fine in Backtest but not Live:  31 Mar 2015, 22:40


joeatbayes said:

I have been using limit orders for TradeType.Buy successfully for quite some time but am struggling with the .Sell due to a "INVALID_REQUEST" message as shown below.    The same code runs fine in a backtest but fails when running live in a virtual account.   From the cAlgo journal it looks like it sent the order to the server with the right set of parameters or at least about what I would have expected. 

Snippet of Code 

 var orderResult = PlaceLimitOrder(TradeType.Sell, Symbol, orderVolume, targetLimitPrice, StratLabel, stopPips, profitPips, expireTime, cmt);
 

Can anybody see what I am doing wrong?

Error from CAlgo Journal

31/03/2015 06:24:00.857 | Limit order to Sell 0.03 Lot(s) EURUSD (Price: 1.07831, SL: 1.08081, TP: 1.07461, ExpireDate: 31/03/2015 06:24) is sent to server
31/03/2015 06:24:01.029 | ? Limit order to Sell 0.03 Lot(s) EURUSD (Price: 1.07831, SL: 1.08081, TP: 1.07461, ExpireDate: 31/03/2015 06:24) is REJECTED with error "INVALID_REQUEST"

Values from my External Log File

L513: 2015-03-31 06:24:00.826: Bid=1.07831 Ask=1.07838 targetLimitPrice=1.07831  stopPips=24.80113 profitPips=37.20170 Account.FreeMargin=984.62 Account.Equity=984.62 PipSize=0.0001 Symbol.PipValue=0.0001 

March-31 - Trade result is showing "Error: TechnicalError"

I modified the SL and take profit to require a Minimum of 1 Pip in the PlaceLimitOrderCall.  Also converted them to Integer values since that was shown in the examples.  I am now printing out the TradeResult and get the following Error:

 L527: 2015-03-31 18:54:07.628: OrderResult=TradeResult (Error: TechnicalError)

Request:   When the server rejects and order it would be nice to print out all the Order parameters the server thinks it rejected and would be even better if we got an indication of which parameters the server finds unsuitable. 

For the same Trade the Journal is still showing:

31/03/2015 18:54:07.456 | Limit order to Sell 0.03 Lot(s) EURUSD (Price: 1.07406, SL: 1.07646, TP: 1.07036, ExpireDate: 31/03/2015 18:54) is sent to server

31/03/2015 18:54:07.628 | → Limit order to Sell 0.03 Lot(s) EURUSD (Price: 1.07406, SL: 1.07646, TP: 1.07036, ExpireDate: 31/03/2015 18:54) is REJECTED with error "INVALID_REQUEST"

The extract from my Text log file with parameters we are sending in the API call.

L513: 2015-03-31 18:54:07.456: CheckOpenShort: OPEN SELL cmt=Bid=1.07406 Ask=1.07413 
targetLimitPrice=1.07406 stopPips=24.00000 profitPips=37.00000 LimitPriceEMA=1.07406 
VeryShortEMA=1.07417 ShortEMA=1.07423275951391 LongEMA=1.07425 stopLimitPriceAdj=0.002470338 
atRiskPerShare=0.00254444814 MaxCapitalAtRiskForSingleTrade=9.8989 maxShares=3890.39172950092 
totalMaxRisk=7.63334442 orderVolume=3000 stopPips=24 profitPips=37 Account.FreeMargin=989.89 
Account.Equity=989.89 PipSize=0.0001 Symbol.PipValue=0.0001 workingLongStopLimitPort=0.0023

Advice on diagnosing this would be greatly appreciated. 

@Limit Order Failing for TradeType.Sell but not TradeType.Buy "INVALID_REQUEST" Works fine in Backtest but not Live:  31 Mar 2015, 10:17


I have been using limit orders for TradeType.Buy successfully for quite some time but am struggling with the .Sell due to a "INVALID_REQUEST" message as shown below.    The same code runs fine in a backtest but fails when running live in a virtual account.   From the cAlgo journal it looks like it sent the order to the server with the right set of parameters or at least about what I would have expected. 

Snippet of Code 

 var orderResult = PlaceLimitOrder(TradeType.Sell, Symbol, orderVolume, targetLimitPrice, StratLabel, stopPips, profitPips, expireTime, cmt);
 

Can anybody see what I am doing wrong?

Error from CAlgo Journal

31/03/2015 06:24:00.857 | Limit order to Sell 0.03 Lot(s) EURUSD (Price: 1.07831, SL: 1.08081, TP: 1.07461, ExpireDate: 31/03/2015 06:24) is sent to server
31/03/2015 06:24:01.029 | ? Limit order to Sell 0.03 Lot(s) EURUSD (Price: 1.07831, SL: 1.08081, TP: 1.07461, ExpireDate: 31/03/2015 06:24) is REJECTED with error "INVALID_REQUEST"

Values from my External Log File

L513: 2015-03-31 06:24:00.826: Bid=1.07831 Ask=1.07838 targetLimitPrice=1.07831  stopPips=24.80113 profitPips=37.20170 Account.FreeMargin=984.62 Account.Equity=984.62 PipSize=0.0001 Symbol.PipValue=0.0001 

@cAlgo instances:  21 Feb 2015, 01:46


I have nearly identical requirements.   It seems like this should be a feature built into the application.   Otherwise it makes it difficult to run multiple copies with different configurations. 

@Using multiple C# source files while working only inside of cAglo?:  26 Dec 2014, 00:53


I would have some utility classes such as   public class parsedRow which I would like to move into separate C# source members from my main cbot.  I do not have visual studio installed and want to work only inside of calgo.   How do I add a reference to an extra C# file to the cBot directly without having to convert it to a DLL.  EG:   In  Lua I can use require "filename"   in F# I can do something similar.  

@exporting/importing data in real time:  26 Dec 2014, 00:45


see:  /algos/cbots/show/588  and /algos/cbots/show/591  I have also written the portion which does the import and executes trades based on external signals.   If others will find it of value then let me know and I can post it as a new CBot.  It takes quite a bit of extra effort to package things for external consumption but if there is sufficient demand I will do so.  Thanks Joe E. 

@How to connect (C#) ?:  26 Dec 2014, 00:41


I would also like the functionality of referencing  CAlgo as a loaded library rather than running it as a separate application.   For me the more important aspect is the ability to auto-start the system,  auto log in and debug easily all inside of visual studio.     Testing from inside of CAlgo is cumbersome and quite often forces me back to hand writing "Print" statements.    

@Data export to file:  26 Dec 2014, 00:30


See: /algos/cbots/show/591 and /algos/cbots/show/588

Notification Publishing copyrighted material is strictly prohibited. If you believe there is copyrighted material in this section you may use the Copyright Infringement Notification form to submit a claim.

No uploaded.

Warning! Executing cBots downloaded from this section may result in loss of funds. Use them at your own risk.
free  22 Dec 2014
Exports chosen bars for Symbol at chosen at the chosen bar size from calgo to CSV file in Directory specified.      Files can be imported by excel but do not open in  when trying to update as excel takes an exclusive lock which will crash the process. OK: Help Needed:   When I run this in Backtest mode with 1 minute bars I get something that looks like valid volumes.  When I run it with 10 tick bars I get the same volume in every bar.    How do we get valid volume for the smaller duration bars.  Question:  How do you convert (MarketSeries.TickVolume to actual Transaction volume for the Bar. I want Total number of units traded during the bar.   I think transaction volume is a more meaningful input for momentum because a small transaction of a fractional unit doesn't mean as much as if a 100 million $ changed hands during the same change in price.  Recommended used is to use the Backtester to download as much historical data as possible in a back-test then switch over to activating the strategy live.  It will auto-backfill the data between end of Backtest and live data but use the Backtest to get as much of the data as possible first as it's lastBar semantic is simplistic.  I have found that tickvolume is only accurate when the Backtest is using the "Tick Data from Server"       The system is smart enough to add new data from current Backtest to end of existing data and it will not overwrite or duplicate the older data but it is not smart enough to detect gaps if the user runs a Backtest covering a couple of months then skips a few months and runs a different Backtest. When used live it will write a new bar whenever a bar has been completed for as long as the cbot is active.    The files are opened in shared mode so a separate process can read new data from them as they arrive.  I needed this because our AI prediction engine needs new data as it arrives and can not run inside of calgo.  I will eventually  write the reverse plumbing to allow predictions from the external engine to be picked up by calgo and executed as trades. When used in back-tester will export the data in the time-frame specified by the backtest.   I tested it back to Jan-1-2011 and all the data seemed to show up.         File name is derived from the symbol, bar duration and chosen moving average and looks like:  exp-EURUSD-Minute-ma3-bars.b1.csv Data  Looks as follows: datetime,open,close,high,low,volume,weighted_val,rsi,stddev,spread 2014-01-27 00:00:00.000,1.368700,1.368630,1.368730,1.368610,23.000000,1.368650,67.318063,0.000223,0.000050 2014-01-27 00:01:00.000,1.368620,1.368590,1.368630,1.368590,15.000000,1.368600,66.005144,0.000282,0.000050 2014-01-27 00:02:00.000,1.368570,1.368860,1.368880,1.368490,41.000000,1.368773,70.226266,0.000365,0.000060 2014-01-27 00:03:00.000,1.368880,1.368980,1.369010,1.368880,29.000000,1.368963,71.896500,0.000438,0.000050 2014-01-27 00:04:00.000,1.368970,1.368980,1.368980,1.368970,4.000000,1.368978,71.896500,0.000483,0.000030 2014-01-27 00:05:00.000,1.369010,1.368810,1.369010,1.368810,20.000000,1.368860,65.829147,0.000504,0.000070 2014-01-27 00:06:00.000,1.368800,1.368610,1.368810,1.368610,28.000000,1.368660,59.470578,0.000500,0.000080 2014-01-27 00:07:00.000,1.368590,1.368970,1.368980,1.368550,36.000000,1.368868,65.862483,0.000511,0.000060 2014-01-27 00:08:00.000,1.368980,1.368990,1.369000,1.368980,6.000000,1.368990,66.181582,0.000512,0.000060 In Excel it looks like: Note: To auto parse the dates when loading in excel change the code to replace T between dateTime with a space. Enhanced Dec-21-2014: To only add bars newer than what are already present in the file but it is still missing the code  to detect gaps and try to fill them in.   Enhanced Dec-212-2014 - To attempt to backfill bars between current bar and the last bar it finds in the file.  This is needed because the the back tester in CAglo always runs a little behind the current data and would create a gap of some very critical missing current data.     This process calculates how many are missing and attempts to backfile from current series.   I have tested it over a weekend but not over longer periods of times.     Want to collaborate:   www.linkedin.com/pub/joe-ellsworth/0/22/682/  or http://BayesAnalytic.com No Promises,  No Warranty.  
free  23 Dec 2014
Runs as a bot  that writes tick data to a  external CSV file  that can be read by other processes.    I needed this because our AI algorithm exists outside of calgo and needs fresh data every tick to make it's predictions.     I will eventually build the plumbing to allow external buy / sell recommendations to be read back from another stream using a separate bot. For those who want bar data instead of tick data.  Look for DataExportBars. Enhanced 2014-12-21 - Added the num_bid, num_ask and vol_adj_ask, vol_adj_bid columns based on market depth also removed some columns that have values that never changed.   I think  vol_adj_bid and vol_adj_ask show where the market really is with less gaming.  I like this because it shows where the bulk of the real orders are concentrated.   That way the odd sell at a very low or very high price people use to try and game the system are averaged out because they typically do not use large volumes for this kind of gaming.   It may be even better to throw away the statistical outliers and average the rest. Enhanced:2014-12-21 - Reverse spread for vol_adj_price and added a price adjust volume to provide a basic bull vs bear indicator based on where the bulk the open order book is hanging.   When used against backtest it can download historical data.  (See instructions in source) but the Market depth values I use to set num_bid, num_ask, vol_adj_ask, vol_adj_bid are not available during back test which is a real shame because I think these would be really handle for Machine learning algorithms.   The output file name is constructed from the Symbol plus the length of the moving average.  An example is: exp-EURUSD-ma30-ticks.csv   The moving average is included as an additional field just to show how it can be merged with the tick data. Sample Output: date,ask,bid,spread,num_ask,num_bid,vol_adj_ask,vol_adj_bid,vol_adj_spread 2014-12-22 06:51:29.746,1.225520,1.225410,0.000110,11,10,1.225600,1.225347,-0.000252 2014-12-22 06:51:36.527,1.225520,1.225420,0.000100,9,8,1.225590,1.225356,-0.000234 2014-12-22 06:51:44.308,1.225520,1.225430,0.000090,8,10,1.225584,1.225355,-0.000229 2014-12-22 06:51:44.621,1.225520,1.225440,0.000080,7,11,1.225588,1.225357,-0.000231 2014-12-22 06:51:47.621,1.225540,1.225440,0.000100,9,10,1.225594,1.225358,-0.000235 2014-12-22 06:51:49.371,1.225540,1.225460,0.000080,12,9,1.225635,1.225345,-0.000290 2014-12-22 06:51:49.621,1.225550,1.225460,0.000090,12,13,1.225662,1.225332,-0.000330 2014-12-22 06:51:50.371,1.225560,1.225460,0.000100,12,11,1.225675,1.225336,-0.000339 2014-12-22 06:51:50.605,1.225570,1.225460,0.000110,11,13,1.225682,1.225334,-0.000348 2014-12-22 06:51:50.918,1.225550,1.225450,0.000100,11,12,1.225684,1.225337,-0.000348 2014-12-22 06:51:51.168,1.225570,1.225490,0.000080,11,15,1.225667,1.225319,-0.000348 2014-12-22 06:51:51.480,1.225600,1.225520,0.000080,13,12,1.225723,1.225392,-0.000331 2014-12-22 06:51:51.621,1.225620,1.225520,0.000100,11,11,1.225703,1.225413,-0.000290 2014-12-22 06:51:52.418,1.225610,1.225520,0.000090,9,12,1.225676,1.225436,-0.000241 Sample in Excel When ran live can save new inbound ticks to the file as they arrive where a external process can pick them up.   I opened the file in shared mode so both processes can access the file simultaneously. It delivers a poor mans pipe for freshly arriving bar data.    I tested this in Lua and it will sit there waiting for new bars to show up at the end of the file and seems to catch them every time they arrive.     When using in this mode I recommend using a fast SSD or Ramdisk to minimize IO latency to disk. I apologize to those who want to use it internationally since I know the comma delimiter will mess up your formatting.  I tried changing it to a Tab delimited format but then it takes 5 extra clicks to load into excel.  There are only two lines where you have to change the , to tab to make  it work internationally.  I did not try to fix the back fill data problem for ticks like I did with bars because my application did not find them essential.  Let me know if you want this feature we can not duplicate it exactly but should be able to get close except for being able to backfill the gaps between end of back-test and real time.  Note:  To avoid saving ticks that had no order book I added a If Statement if ((volAdjAsk == 0.0) && (volAdjBid == 0.0)) it will have to be commented if you want to use it to download historical bars.  Want to collaborate:     www.linkedin.com/pub/joe-ellsworth/0/22/682/ or     http://bayesanalytic.com No Promises,  No Warranty.  Terms of use MIT http://opensource.org/licenses/MIT