cAlgo keeps on disconnecting when using HFT algo.

09 Sep 2014, 14:16cAlgo keeps on disconnecting when using HFT algo.#1
jobenbposts: 103since: 07 Oct 2013

Hi cAlgo Team

I am trading a few High Frequency Algorithms through cAlgo and I keep on being disconnected. My take profit falls in the 1 - 10 pip range, however I place the orders almost every 10 seconds maybe 10 Limit order positions at a time. This leads to hundreds of positions open and an extremely slow platform with constant connection issues while my other cAlgo accounts on the same computer does not experience it at all. This makes me believe that the client platform or the server implements some form of throttling and disconnecting the client if the client place a lot of orders in a short amount of time.

1. Is this true?

2. If yes, why does it occur and when?

3. Is this a potential issue with cAlgo platform not being able to handle such load?

4. Why are there limitations on the amount of open orders? Will this be changed to be more or less in the future?

 

Thank you.

09 Sep 2014, 14:27#2
cTrader Teamposts: 3380since: 23 Sep 2013

This makes me believe that the client platform or the server implements some form of throttling and disconnecting the client if the client place a lot of orders in a short amount of time.

cServer doesn't have any throttling logic.

We prepared several performance optimizations in cAlgo 1.25. If it is possible please check how your cBots will perform in cAlgo 1.25. Currently cAlgo 1.25 is deployed for Spotware environment only:

http://spotware.ctrader.com/calgo-spotware-setup.exe


TRADERS FIRST™ Vote for your favorite features: http://vote.spotware.com/
10 Sep 2014, 12:52#3
jobenbposts: 103since: 07 Oct 2013

Hi

I have used the Spotware cAlgo 1.25 Yesterday in parallel with the broker cAlgo 1.24 (which continuously kept on disconnecting - with the little red dot next to the account number) and although the latest cAlgo did perform much better it still crashed early this morning unfortunately. I have started it again and will keep it running, so you guys should easily be able to get the load on your side. :-)

How sure can we be that this issue really has been resolved in cAlgo 1.25 and that it is not a server load issue because of the fact that your Spotware server may just be able to handle everything as it is maybe not a true reflection of the load my broker is experiencing?

I do wonder about cServer and if it is not maybe that my broker's cServer is under heavy load not just from my algo but also having thousands of other connections trading at the same time and they decide to do a periodic kill switch on my account due to it causing a lot of the trading order load every few seconds???

Are you guys doing heavy load testing on both the platform and cServer?

Why are there a limit of 2000 orders / positions within cAlgo per account? This limit full scale rollout of algorithms across currency pairs. Could this be increased to 10000 please?

What is the behaviour of cServer if it has too many orders coming in than it can handle?

Please explain the limitations with regards to cAlgo interacting with cServer, so that I can code with that in mind? 

Many thanks and I look forward to your reply!

10 Sep 2014, 14:24#4
cTrader Teamposts: 3380since: 23 Sep 2013

Yes, we do load testing for both client and server side.

cServer has a limitation for the amount of open positions per account: 2000. We plan to increase this limit in the future. If cServer cannot handle all incoming requests it puts them to the queue. There is no any logic to disconnect clients in case of heavy load.

Please send to engage@spotware.com your account number and broker name. We will troubleshoot the issue.

 

 


TRADERS FIRST™ Vote for your favorite features: http://vote.spotware.com/
10 Sep 2014, 15:42RE:#5
jobenbposts: 103since: 07 Oct 2013

Spotware said:

Yes, we do load testing for both client and server side.

cServer has a limitation for the amount of open positions per account: 2000. We plan to increase this limit in the future. If cServer cannot handle all incoming requests it puts them to the queue. There is no any logic to disconnect clients in case of heavy load.

Please send to engage@spotware.com your account number and broker name. We will troubleshoot the issue.

 

 

Thank you. I have sent an email to you containing the details. :-)

I was also thinking about the connecting / disconnecting and I think on the platform side it could also be a timeout that cause the disconnect and then retries after about 3 minutes. I think the timeout might be too short for connection when a lot of orders / positions are involved due to the busyness of the system and context switching of the threads. Is the timeout configurable?

Thank you for all your help! It is much appreciated and I hope we can improve the next release at the same time. ;-)

11 Sep 2014, 10:52RE:#6
jobenbposts: 103since: 07 Oct 2013

Spotware said:

Yes, we do load testing for both client and server side.

cServer has a limitation for the amount of open positions per account: 2000. We plan to increase this limit in the future. If cServer cannot handle all incoming requests it puts them to the queue. There is no any logic to disconnect clients in case of heavy load.

Please send to engage@spotware.com your account number and broker name. We will troubleshoot the issue.

 

 

Just to give you another update. I have had another unhandled exception today on cAlgo version 1.25.

I was also thinking about the connecting / disconnecting and I think on the platform side it could also be a timeout that cause the disconnect and then retries after about 3 minutes. I think the timeout might be too short for connection when a lot of orders / positions are involved due to the busyness of the system and context switching of the threads. Is the timeout configurable?

I look forward to your answers and feedback. Thank you!!

12 Sep 2014, 00:5911/09/2014 19:49:12.649 | cBot crashed: Error #65140095#7
jobenbposts: 103since: 07 Oct 2013

Hi cAlgo Team

I have just had a look at the cAlgo 1.25 platform to verify if it is still running and it seems all the robots have crashed with the following error: 11/09/2014 19:49:12.649 | cBot crashed: Error #65140095

Could you please explain what it means?

I also experience connecting and disconnecting issues with cAlgo 1.25 now, but closing and restarting the platform seems to resolve it again for the next few hours. With cAlgo 1.25 it seems that it only happens once in awhile compared to cAlgo 1.24 with which it happens constantly unless it is during the quiet hours of the market.

Any ideas? Have you had time to verify whether it is a potential timeout which is too short for a high order load (which constantly changes due to TP being reached and new orders being placed)?

Many thanks!!

12 Sep 2014, 15:14#8
cTrader Teamposts: 3380since: 23 Sep 2013

I have just had a look at the cAlgo 1.25 platform to verify if it is still running and it seems all the robots have crashed with the following error: 11/09/2014 19:49:12.649 | cBot crashed: Error #65140095

Could you please specify a scenario to reproduce the problem? Did you remove the account while robot was running?

I also experience connecting and disconnecting issues with cAlgo 1.25 now, but closing and restarting the platform seems to resolve it again for the next few hours. With cAlgo 1.25 it seems that it only happens once in awhile compared to cAlgo 1.24 with which it happens constantly unless it is during the quiet hours of the market.

Any ideas? Have you had time to verify whether it is a potential timeout which is too short for a high order load (which constantly changes due to TP being reached and new orders being placed)?

We will investigate the problem. Thank you for reporting that.


TRADERS FIRST™ Vote for your favorite features: http://vote.spotware.com/
13 Sep 2014, 04:56RE:#9
jobenbposts: 103since: 07 Oct 2013

Spotware said:

I have just had a look at the cAlgo 1.25 platform to verify if it is still running and it seems all the robots have crashed with the following error: 11/09/2014 19:49:12.649 | cBot crashed: Error #65140095

Could you please specify a scenario to reproduce the problem? Did you remove the account while robot was running?

I also experience connecting and disconnecting issues with cAlgo 1.25 now, but closing and restarting the platform seems to resolve it again for the next few hours. With cAlgo 1.25 it seems that it only happens once in awhile compared to cAlgo 1.24 with which it happens constantly unless it is during the quiet hours of the market.

Any ideas? Have you had time to verify whether it is a potential timeout which is too short for a high order load (which constantly changes due to TP being reached and new orders being placed)?

We will investigate the problem. Thank you for reporting that.

Hi,

I did not close any account while any cBot was running. Normally the platform crashed completely, however this time the cBot stopped running and by investigating the log it showed Error #65140095. This could also potentially be related (I think) to the regular connecting and disconnecting issue being experienced when there are a very high open order load with a lot of open positions.

I think this is quite critical to be resolved as it clearly place a hidden barrier for anyone's cBot to place a lot of orders on various instruments. So someone with a quiet cBot will probably never experience this issue but someone like me with advanced medium - high frequency algorithms will not be able to use the platform at all due to these unknown hidden capacity risks. Luckily I picked this up on a demo account, and luckily you guys are determined to fix it! :D

Thank you.

17 Sep 2014, 11:46RE:#10
jobenbposts: 103since: 07 Oct 2013

Spotware said:

I have just had a look at the cAlgo 1.25 platform to verify if it is still running and it seems all the robots have crashed with the following error: 11/09/2014 19:49:12.649 | cBot crashed: Error #65140095

Could you please specify a scenario to reproduce the problem? Did you remove the account while robot was running?

I also experience connecting and disconnecting issues with cAlgo 1.25 now, but closing and restarting the platform seems to resolve it again for the next few hours. With cAlgo 1.25 it seems that it only happens once in awhile compared to cAlgo 1.24 with which it happens constantly unless it is during the quiet hours of the market.

Any ideas? Have you had time to verify whether it is a potential timeout which is too short for a high order load (which constantly changes due to TP being reached and new orders being placed)?

We will investigate the problem. Thank you for reporting that.

 

 

Dear Spotware

Any luck with finding and resolving the issue?

Thank you.