fbpx
Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1

TOPIC:

DBF Append Speed in X# vs VO - Huge Difference 15 Sep 2020 07:11 #15878

  • Jamal


  • Posts: 241
  • Hi Dev Team!

    I created a small app that collects data from a webservice and adds 100 records every 3 minutes. The file reached millions of records and the .NET app started to slow down (I thought it was freezing since there was no indication what is going behind the scene), then after extensive CPU profiling and debugging, I isolated the issue to a single function in the DBFCDX RDD.

    The X# DBF DBAppend function (or dbSever Append method - also tested) is very, very slow compared the blazing speed in VO. Also, when I used the Advantage AXDBFCDX, it was very fast!

    To test I created a dummy dbf with one field ( Char(20) ) and added 1 million records.
    On my system, X# took 63.27 minutes while VO only took 3.86 seconds. Simpy call the DBAppend function in a loop.

    Using X# public build: 2.5b
    VO 2.8 - 2838
    Windows 10 Pro, SSD drive.

    Here is the code:
    FUNCTION Start() AS VOID
    	LOCAL cDBF AS STRING 
    	LOCAL aFields AS ARRAY 
    	LOCAL i AS DWORD	
            LOCAL nStart, nElapsed AS FLOAT
        
    	RddSetDefault ( "DBFCDX" ) 
    	
    	cDbf := "C:\test\mytest"    // change the dbf to "C:\test\mytestVO"  to test in VO 2.8.
    	FErase ( cDbf + ".cdx" )
    	
    	aFields := { { "LAST" , "C" , 20 , 0 } } 
    	
    	
    	DbCreate( cDBF , AFields)
    	DbUseArea( ,,cDBF )		
        
            nStart := Seconds()
        
           ? "start : " + ntrim(nStart)
        
    	FOR i := 1 UPTO 1000000
    		DbAppend() 		
    	NEXT
    	
           nElapsed := Seconds() - nStart
    
           ? "Elapsed: " + NTrim(nElapsed) + " seconds"
       
    	DbCloseArea()
    		
        wait
    RETURN

    Please Log in or Create an account to join the conversation.

    Last edit: by Jamal.

    DBF Append Speed in X# vs VO - Huge Difference 15 Sep 2020 08:33 #15879

    • robert
    • robert's Avatar


  • Posts: 2222
  • Jamal,
    We are aware of this and we are working on it:
    There is a glitch in the page splitting algorithm .
    The resulting index is correct but contains many pages with only one key per page.
    This seems to be related to the large number of duplicate key values (in your case 1000000 blank keys).

    Robert
    XSharp Development Team
    The Netherlands

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 15 Sep 2020 08:57 #15880

    • Chris
    • Chris's Avatar


  • Posts: 2566
  • Robert,

    I think this is a different issue, because there is no index file involved. Something seems to make appending records much slower in the .Net version.
    XSharp Development Team
    chris(at)xsharp.eu

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 15 Sep 2020 19:16 #15885

    • Jamal


  • Posts: 241
  • Robert,

    In the sample I provided I am just appending simple records; they are not duplicates. If I create an index and use FIELDPUT( ) in the LAST field, the process in X# is extremely slow.

    Please see the modified sample code below.It took a whopping 57 minutes in X#, while VO only took 3.49 seconds!


    So there are two issues, Append and the page splitting algorithm you mentioned:
    FUNCTION Start() AS VOID
    	LOCAL cDBF AS STRING 
    	LOCAL aFields AS ARRAY 
    	LOCAL i AS DWORD
    	
            LOCAL nStart, nElapsed AS FLOAT
        
    	RddSetDefault ( "DBFCDX" ) 
    	
    	cDbf := "C:\test\mytest" 
    	FErase ( cDbf + ".cdx" )
    	
    	aFields := { { "LAST" , "C" , 20 , 0 } } 
    	
    	
    	DbCreate( cDBF , AFields)
    	DbUseArea( ,,cDBF )	
        
            DBCreateOrder("last", "c:\test\mytest", "last")
        
           nStart := Seconds()
        
           ? "start : " + ntrim(nStart)
        
    	FOR i := 1 UPTO 1000000
    		DbAppend() 		
                    fieldput(1, asstring(i))
    	NEXT
    	
           nElapsed := Seconds() - nStart
    
           ? "Elapsed: " + NTrim(nElapsed) + " seconds"
    
        
    	DbCloseArea()
    		
        wait
    RETURN

    Please Log in or Create an account to join the conversation.

    Last edit: by Jamal.

    DBF Append Speed in X# vs VO - Huge Difference 16 Sep 2020 10:10 #15888

    • robert
    • robert's Avatar


  • Posts: 2222
  • Jamal,
    We'll look at this asap. Not for the 2.6 build that is coming in a few days, but in the build after that.
    I know that VO used very aggressive caching for files opened exclusively, like in your example.
    We'll try to build something similar in X#.

    Robert
    XSharp Development Team
    The Netherlands

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 17 Sep 2020 01:20 #15889

    • Jamal


  • Posts: 241
  • Robert,

    Will test again when you issue the fix and I'll report back.

    Thanks!

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 17 Sep 2020 13:35 #15899

    • ic2


  • Posts: 995
  • This seems to be related to the large number of duplicate key values (in your case 1000000 blank keys).

    We were actually the X# users having that issue that many duplicate values corrupted the CDX after mostly 1 or 2 days of use in a CGI webprogram we fully converted to X#. So I am surprised that this issue still exists as I thought it was solved.

    However, as we had another CDX corruption issue and as much as we would have preferfed using the native X# ADS/DBF drivers, we couldn't let the client keep functioning as guinea pig :Pwe eventually reverted the DBFCDX system back to the Vulcan method. Since then the X# webprogram works fine.

    So I was wondering Jamal: do you have the same speed issue if you use the Vulcan DLL/RDD's (only for the DBFCDX functionality)?

    Dick

    Please Log in or Create an account to join the conversation.

    Last edit: by ic2.

    DBF Append Speed in X# vs VO - Huge Difference 17 Sep 2020 18:17 #15902

    • Jamal


  • Posts: 241
  • Dick,

    I don't have Vulcan anymore and so I did not test it.

    Jamal

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 22 Sep 2020 10:07 #15941

    • KeesIC2
    • KeesIC2's Avatar


  • Posts: 16
  • I think all the changes and improvements in 2.6 are less important than a reliable DBF/CDX system. I can't use X# because indexes get corrupted even when using only basic functionality. Yet more and more new features are added while this is just not working properly.

    Please Log in or Create an account to join the conversation.

    Last edit: by KeesIC2.

    DBF Append Speed in X# vs VO - Huge Difference 22 Sep 2020 10:40 #15943

    • robert
    • robert's Avatar


  • Posts: 2222
  • Kees,
    We have already made some changes in this area, so most likely the problem is fixed.
    Did you test the new version?

    Robert
    XSharp Development Team
    The Netherlands

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 01 Jun 2021 19:01 #18701

    • Jamal


  • Posts: 241
  • Robert,

    Just wanted to let you know that the DBF speed issue persists in X# 2.8a.

    Jamal

    robert wrote: Jamal,
    We'll look at this asap. Not for the 2.6 build that is coming in a few days, but in the build after that.
    I know that VO used very aggressive caching for files opened exclusively, like in your example.
    We'll try to build something similar in X#.

    Robert

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 01 Jun 2021 22:09 #18702

    • ic2


  • Posts: 995
  • Hello Jamal,

    That is quite frustrating. We have a few programs, 1 exe and the rest in DLL's, using DBF's, but these are not data intensive.

    I wonder if the speed issue also comes up when using ADS? Do you happen to have a test program that compares VO & X#? Then I can run it using X# native ADS access and with the Vulcan RDD as well and give you the results, if you can publish the code.

    We can not afford to move to X# if there's a considerable append speed issue

    Dick

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 01 Jun 2021 22:16 #18703

    • Jamal


  • Posts: 241
  • Dick,

    Just use the code I posted above (direct link): www.xsharp.eu/forum/public-product/2146-...uge-difference#15885

    ic2 wrote: Hello Jamal,

    That is quite frustrating. We have a few programs, 1 exe and the rest in DLL's, using DBF's, but these are not data intensive.

    I wonder if the speed issue also comes up when using ADS? Do you happen to have a test program that compares VO & X#? Then I can run it using X# native ADS access and with the Vulcan RDD as well and give you the results, if you can publish the code.

    We can not afford to move to X# if there's a considerable append speed issue

    Dick

    Please Log in or Create an account to join the conversation.

    Last edit: by Jamal.

    DBF Append Speed in X# vs VO - Huge Difference 02 Jun 2021 01:15 #18704

    • Chris
    • Chris's Avatar


  • Posts: 2566
  • Hi Jamal,

    Indeed, there still seems to be a problem, in VO the time needed/no of records is always a linear function, while in X# speed decreases exponentially after around 100,000 records.

    For smaller number of total records, X# speed is comparable to that of VO, although of course it can never become quite as fast, due to the overhead of .Net.
    XSharp Development Team
    chris(at)xsharp.eu

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 02 Jun 2021 01:25 #18705

    • Jamal


  • Posts: 241
  • Hi Chris,

    That's what my testing also showed.

    This becomes a real issue when doing database bulk operations such importing data or updating existing rows.

    I wonder if Vulcan.NET has such an issue.

    Jamal

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 02 Jun 2021 01:41 #18706

    • Chris
    • Chris's Avatar


  • Posts: 2566
  • Hi Jamal,

    No, vulcan seems it does not suffer from this exponential slow down. It is also quite faster in general, but then again the CDX RDD in vulcan is written in managed c++, which makes a big difference. I am sure it will be further improved also in X#, it's just the usual problem of having only 24 hours in a day... Still, RDD speed in X# has improved a lot in the last builds, in different scenarios than that of your sample.

    Btw, to answer you question Dick, dbf speed with ADS should not have a difference between X# and VO, since all the work is done in the server by ADS itself.
    XSharp Development Team
    chris(at)xsharp.eu

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 03 Jun 2021 23:22 #18720

    • Jamal


  • Posts: 241
  • Hi Chris,

    Despite the fact I reported this issue last September, I can appreciate the constraint of time, however, I hope the dev team improves this situation to almost a few milliseconds (or seconds) of VO speed.

    Jamal

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 04 Jun 2021 01:37 #18721

    • Chris
    • Chris's Avatar


  • Posts: 2566
  • Hi Jamal,

    I am afraid this will be totally impossible. Even vulcan, in which the RDD source code is an almost 1-1 copy of the VO code, compiled in managed c++ with max level of optimizations, does not come anywhere close to that, due to the .Net overhead.

    .Net in so many things is far superior to Win32, but when it comes to very low level code, it has a clear performance disadvantage by definition, so I am afraid we have to live with the fact that dbf usage will be slower. Having said that, since you (and others) have reported those speed issues, performance has already actually improved a lot. But I agree there' still enough room of improvement, especially in this case where something is causing this exponential increase in needed time to perform the operations.
    XSharp Development Team
    chris(at)xsharp.eu

    Please Log in or Create an account to join the conversation.

    Last edit: by Chris.
    • Page:
    • 1