Welcome, Guest |
Please use this forum to post questions about Visual Objects and Vulcan.NET
|
TOPIC:
VO 32 Bit limitations 11 Jul 2018 12:06 #5411
|
With our always growing VO program for certain power users we quickly reach the 2 GB of the data segment for 32Bit programs, so that the program stops with "not enough memory" or similar messages. Of course we with the transition to Dotnet some of this behavior will be better, but we are still about at least 3/4 year away from going to X# programs for our customers. One part is the the VO dynamic memory that we use. We need between 96 MB and 256 MB of dynamic memory that is doubled in the execution. But the main reason for the large usage of data is the memory consumation only because of loading dlls. I have made an analysis with the sysinternals program VMMAP for one dll of media size containing 349 classes, mainly windows. Loading this dll without creating an object uses already 23MB of data. Having loaded about 40 dlls this summarizes already to about 700 MB in the data segment, mainly "Image" and "Heap memory". I tried to modify the linker settings in the project settings, but this has not much influence. Removing references to Dlls helps a little bit, especially the GUI classes and dlls that include the GUI classes need much memory. Removing seldom classes to other Dlls helps, but this is much work, because it needs hundreds of classes moved to see a difference. Does anybody has an idea for optimizations? (Please do not answer that removing code will help....) Thank you Arne Attachments: |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 11 Jul 2018 12:16 #5412
|
Hi Arne, I do not have a solution for you, unfortunately - only a confirmation. When you load a VO DLL, all classes that are contained are loaded in memory. I had asked Uwe Holz many, many years ago if that could be eventually changed, and his answer was that this would destroy the base on which the entire VO runtime system was built. Therefore I see the only possibility for you to reduce the number of classes in one application, maybe to split the application in parts, or maybe load the accounting part only when needed, hoping that the same person does not needs accounting and stock managment at the same time and in the same running copy. For sure, this is not an easy job, and maybe it could make a transition to X# a bit more urgent. Wolfgang |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 11 Jul 2018 19:01 #5420
|
Hi Arne, Robert posted the following on Feb 2, 2017 to xsharp.public.vovulcan newgroups (not sure if it is still running).
METHOD CheckDynMemSize() CLASS MyShell
LOCAL nActualSizeAllocated AS DWORD
LOCAL nActualSizeused AS DWORD
LOCAL nActualSizeMax AS DWORD
LOCAL nMaxDynAllowed AS DWORD
LOCAL nMaxPages AS DWORD
LOCAL nNewSizeInPages AS DWORD
LOCAL nNewSizeInBytes AS DWORD
LOCAL nUsed, nMax, nCheck AS DWORD
//STATIC aIamUsingMemory:= {} as Array
STATIC lBusy AS LOGIC
IF ! lBusy
lBusy := TRUE
// Check RegisterAxit
nUsed := Memory(MEMORY_REGISTERAXIT)
nMax := SetMaxRegisteredAxitMethods(0)
nCheck := DWORD(MulDiv(LONG(nMax), 90, 100))
//DebOut("RegisterAxit", nUsed, nCheck, nMax)
IF nUsed > nCheck // > 90%
DebOut("Approaching RegisterAxit limit, COLLECTFORCED")
Sleep(2000)
CollectForced()
nUsed := Memory(MEMORY_REGISTERAXIT)
IF nUsed > nCheck
SetMaxRegisteredAxitMethods(DWORD(nMax * 1.5)) // 50% more
ENDIF
ENDIF
// Check RegisterKid
nUsed := Memory(MEMORY_REGISTERKID)
nMax := SetMaxRegisteredKids(0)
nCheck := DWORD(MulDiv(LONG(nMax), 90, 100))
//DebOut("RegisterKid", nUsed, nCheck, nMax)
IF nUsed > nCheck // > 90%
DebOut("Approaching RegisterKid limit, COLLECTFORCED")
Sleep(2000)
CollectForced()
nUsed := Memory(MEMORY_REGISTERKID)
IF nUsed > nCheck
SetMaxRegisteredKids(DWORD(nMax * 1.5)) // 50% more
ENDIF
ENDIF
// Check KidStackSize
nUsed := Memory(MEMORY_STACKKID)
nMax := SetKidStackSize(0)
nCheck := DWORD(MulDiv(LONG(nMax), 90, 100))
//DebOut("KidStackSize", nUsed, nCheck, nMax)
IF nUsed > nCheck // > 90%
DebOut("Approaching KidStack limit, COLLECTFORCED")
Sleep(2000)
CollectForced()
nUsed := Memory(MEMORY_STACKKID)
IF nUsed > nCheck
SetKidStackSize(DWORD(nMax * 1.5)) // 50% more
ENDIF
ENDIF
nMaxDynAllowed := 1024*1024*256
nMaxPages := nMaxDynAllowed /0x10000
nActualSizeAllocated := DynInfoSize() * 0x10000 // DyInfoSize returns the number of 64Kb pages
nActualSizeused := DynInfoUsed()
nActualSizeMax := SetMaxDynSize(0)
// When we have used 80 % or more of the allocated memory, then it is time to allocate some more pages
IF nActualSizeused > nActualSizeAllocated * 0.80
CollectForced()
nActualSizeused := DynInfoUsed()
// Debout("MaxAllowed ",nMaxDynAllowed, nMaxPages)
// Debout("MaxDynSize ",nActualSizeMax, nActualSizeMax/0x10000)
// Debout("DynSizeUsed",nActualSizeUsed, nActualSizeUsed/0x10000)
// Debout("Allocated ",nActualSizeAllocated, DynInfoSize())
IF nActualSizeused > nActualSizeAllocated * 0.80
nNewSizeInPages := DynInfoSize() +32 // 32 pages = 2 Mb pages extra
nNewSizeInBytes := nNewSizeInPages * 0x10000
//Debout("Allocating new dynamic memory, old size (in pages) ",DynInfoSize(), " new size ",nNewSizeInPages)
// Make sure that we do not allocate more then the DynInfoMax. If we reach that, then we need to set the max to a higher number
IF nNewSizeInBytes >= nActualSizeMax
IF nNewSizeInPages >= nMaxPages
// MessageBox(NULL_PTR, Cast2Psz("The program is using a lot of memory at this moment. This will have a negative effect on the performance. " + CRLF+;
// "Please close some open windows when possible." +CRLF+ ;
// "When you see this message repeatedly then please restart the program"), Cast2Psz("Program running low on memory"), 0)
//Debout("Resize Max Dyn to ", nMaxDynAllowed, nMaxDynAllowed/0x10000)
SetMaxDynSize(nMaxDynAllowed) // Allocate some extra memory just to be sure
ELSE
//Debout("Resize Max Dyn to ", nNewSizeInBytes, nNewSizeInBytes/0x10000)
SetMaxDynSize(nNewSizeInBytes) // Allocate some extra memory just to be sure
DynSize(nNewSizeInPages-1)
ENDIF
ELSE
//Debout("Allocate", nNewSizeInPages)
DynSize(nNewSizeInPages)
ENDIF
ENDIF
ENDIF
lBusy := FALSE
ENDIF
RETURN NIL
Jamal |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 12 Jul 2018 07:57 #5422
|
Hi Jamal, thank you for sending me the code. I have implemented these functions with the only difference that the parameters are set once at the beginning of the program. The dynamic memory does not give many problems so. Most of the cases the errors happen when the program tries to load another DLL dynamically. I will add a collectforce before loading the Dlls. Probably it may help a little bit. > many moons ago Yes, right, I have stopped counting the moons but it must be many, many ![]() Arne |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 12 Jul 2018 09:48 #5423
|
Arne, CollectForced() before loading a DLL will not make a difference. It moves things around in already allocated memory but will not free any memory. The problem with loading DLLs later 'on demand' is that the OS most likely cannot find a hole in memory with enough space to fit the DLL. Memory will then be used by: - The EXE and Other DLLs loaded into memory - Static Memory allocated by your app and the runtime (for example the runtime allocates a block of memory to store the list of classes and methods) - Dynamic memory allocated by your app. To be more precise: Dynamic memory reserved by your app. Your app by default reserves 2 dynamic memory pages of the size of MaxDynSpace(). The default for this is 16Mb for each page. This memory gets allocated 'when needed'. However the address space is reserved immediately. DynInfoSize() tells you how much of this memory has actually already been allocated. You can force the runtime to physically allocate the memory with DynSize(). If you want to improve things you could consider: - Splitting the dynamically loaded DLLs into smaller ones. Then it might be easier for the OS to find a hole in memory where it fits - Loading DLLs earlier , so allocated dynamic and static memory will not have taken the 'large spots'. Robert XSharp Development Team The Netherlands |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 12 Jul 2018 10:14 #5425
|
Robert, thank you for answering. I did not think on the the possible fragmentation of the memory. This is another possibility for our errors apart from the 2 GB. > - Splitting the dynamically loaded DLLs into smaller ones. Then it might be easier for the OS to find a hole in memory where it fits We have already split the program into many dlls and it seems that it has sometimes the disadvantage of adding overhead for loading every dll. Loading 80 of all 108 existing Dlls plus the Crystal Reports runtime engine for example normally is possible. But it means that the Virtual size has reached already over 1,5 GB without having opened many windows. Normally about 40 dlls are loaded. The program items themselves also need memory. To avoid problems with the garbage collector we store much data using Memalloc, ... for example when reordering tables in memory. Of course this can be a source of fragementation. Some time ago I found a memory leak and I am still searching for possible places, but could not find any until now. Additionally ODBC seems to take memory for every open cursor and in a variable amount. Arne |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 12 Jul 2018 11:15 #5429
|
I have made some memory analysis looking into a productive program of a a colleague on a terminal server with the program VMMAP without letting him know ![]() - VMMapfrag.png - Main parts of the memory usage are: - total 1894MB) - Heap 820MB - Private Data (yellow= VO Dynamic Memory) 560 MB - Image (DLL code) 382 MB - Free space left: 216 MB The fragmentation does not look so bad: The VO dynamic memory is allocated in once. The heap is allocated by blocks of 16M, 8M, 4 M, 2 M, ... The Dll image memory is in between. There is few unused space between the different places, -VMMapimage.png This shows the space used by the Dlls ordered by Size desc. (What is interesting that the DLLs occupy space in the data segment, originally I learned that there are 2 GB address space and 2 GB data space for a 32 Bit prog). Most of the Dlls occupy space beween 2 and 10 MB, few take more space. -VMMapHeap1.png This shows the heap distribution ordered by Size desc. I have not found out yet, where many of these blocks come from. There are many blocks that contain strings like the block examined with these texts JPDB, ATAD, ... -VMMapLater.png This is now 15 Minutes later - Main parts of the memory usage are: - total 1970 MB - Heap 877 MB - Private Data 562 MB - Image (DLL code) 382 MB - Free: 140 MB So no dlls have been loaded, the heap has grown and the shareable memory has grown a little bit Arne Attachments: |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 12 Jul 2018 11:51 #5430
|
Hi Arne, AFAIK there should be a possbility to increase the memory of a 32 bit application until 4 GB. I don't know if this one solves the problem: www.maketecheasier.com/increase-memory-l...n-windows-64-bit-os/ ntcore.com/?page_id=371 Wolfgang |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 12 Jul 2018 12:33 #5434
|
Great Wolfgang! Everything still seems to work including the Crystal Reports. I have now in my test program a total of 2642 MB and it still shows free memory 1612 MB! 82 Dlls loaded, 520 SQL Statements opened, 480 SQL-Server objects, 25 Windows with many tab windows each one, nearly incredible! Now I get come to the limits of the user/gdi objects of the process and the program gets slow because of the garbage collector, but this is not a problem currently. (By the way: In my tests also the Dotnet programs get much slower with much possible garbage to collect, but they perform still much better than the VO programs) There some characters changed in the executable by the patch program. It is a nice solution. For 32-Bit programs until now I had only found descriptions about using 3 GB instead of 2 GB, but always with advices better not to do it. I will make further tests the next weeks and let you know. Many thanks Arne |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 12 Jul 2018 21:13 #5451
|
Hi Wolfgang, That's an interesting find! I ran it on an EXE and now see available memory more than doubled. While we are at it, i did some search on the web and found out that we can use EDITBIN.EXE from the "Visual Studio developer command prompt". It is part VC++ runtime installation. EDITBIN /LARGEADDRESSAWARE Your32bitEXE Refer to: stackoverflow.com/questions/1346480/how-...-large-address-aware docs.microsoft.com/en-us/cpp/build/reference/editbin-options BTW: editbin.exe is also part of MASM32 at www.masm32.com. After installation it will be in BIN folder. It is old, but it does the same thing. Thanks! Jamal |
Please Log in or Create an account to join the conversation. Last edit: by Jamal. |
VO 32 Bit limitations 14 Jul 2018 07:03 #5478
|
I found some C# code that checks to see if an EXE is LARGE_ADDRESS_AWARE, and if not , it will update its header. I've verified that it works by using VMMAP. I created a C# console app which is run as a part of my installation program and updates the EXE files found in installation folder as arguments. Anyway, I do not have to rely on third party tools except the .NET framework. ![]() Source: stackoverflow.com/questions/9054469/how-...addressaware/9056757 Credit goes to authors: static bool LargeAware(string file) {
using (var fs = File.Open(file, FileMode.Open, FileAccess.ReadWrite, FileShare.None)) {
bool b = LargeAware(fs);
fs.Close();
return b;
}
}
const int IMAGE_FILE_LARGE_ADDRESS_AWARE = 0x20;
static bool LargeAware(Stream stream) {
var br = new BinaryReader(stream);
var bw = new BinaryWriter(stream);
if (br.ReadInt16() != 0x5A4D) //No MZ Header
return false;
br.BaseStream.Position = 0x3C;
var peloc = br.ReadInt32(); //Get the PE header location.
br.BaseStream.Position = peloc;
if (br.ReadInt32() != 0x4550) //No PE header
return false;
br.BaseStream.Position += 0x12;
long nFilePos = (int)br.BaseStream.Position;
Int16 nLgaInt = br.ReadInt16();
bool bIsLGA = (nLgaInt & IMAGE_FILE_LARGE_ADDRESS_AWARE) == IMAGE_FILE_LARGE_ADDRESS_AWARE;
if (bIsLGA)
return true;
nLgaInt |= IMAGE_FILE_LARGE_ADDRESS_AWARE;
long nFilePos1 = bw.Seek((int)nFilePos, SeekOrigin.Begin);
bw.Write(nLgaInt);
bw.Flush();
long nFilePos2 = br.BaseStream.Seek(nFilePos, SeekOrigin.Begin);
nLgaInt = br.ReadInt16();
bIsLGA = (nLgaInt & IMAGE_FILE_LARGE_ADDRESS_AWARE) == IMAGE_FILE_LARGE_ADDRESS_AWARE;
return bIsLGA;
} |
Please Log in or Create an account to join the conversation. Last edit: by Jamal. |
VO 32 Bit limitations 14 Jul 2018 07:38 #5479
|
Here the X# translation of the C# code as a console app. Tested and works as expected ![]() USING System
USING System.Collections.Generic
USING System.Linq
USING System.Text
USING System.IO
BEGIN NAMESPACE XSharpMakeLargeAddressAware
#DEFINE IMAGE_FILE_LARGE_ADDRESS_AWARE 0x20
FUNCTION Start() AS VOID
MakeLargeAware("c:\somefolder\programname.exe") // replace with your own exe path
return
function MakeLargeAware(file as string) as logic
LOCAL b AS LOGIC
begin using var fs := File.Open(file, FileMode.Open, FileAccess.ReadWrite, FileShare.None)
b := LargeAware(fs)
fs.Close()
end using
return b
function LargeAware(stream as Stream) as logic
var br := BinaryReader{stream}
var bw := BinaryWriter{stream}
if (br.ReadInt16() != 0x5A4D) //No MZ Header
return false
ENDIF
br:BaseStream:Position := 0x3C
var peloc := br:ReadInt32() //Get the PE header location.
br:BaseStream:Position := peloc
if (br.ReadInt32() != 0x4550) //No PE header
return false
ENDIF
br:BaseStream:Position += 0x12
local nFilePos := (int)br:BaseStream:Position as longint
local nLgaInt := br:ReadInt16() as SHORT
local bIsLGA := (_And(nLgaInt, IMAGE_FILE_LARGE_ADDRESS_AWARE) == IMAGE_FILE_LARGE_ADDRESS_AWARE) AS LOGIC
if (bIsLGA)
RETURN TRUE
ENDIF
nLgaInt |= IMAGE_FILE_LARGE_ADDRESS_AWARE
local nFilePos1 := bw:Seek((int)nFilePos, SeekOrigin.Begin) as int64
bw.Write(nLgaInt)
bw.Flush()
local nFilePos2 := br:BaseStream:Seek(nFilePos, SeekOrigin.Begin) as int64
nLgaInt := br:ReadInt16()
bIsLGA := (nLgaInt & IMAGE_FILE_LARGE_ADDRESS_AWARE) == IMAGE_FILE_LARGE_ADDRESS_AWARE
return bIsLGA
END NAMESPACE
|
Please Log in or Create an account to join the conversation. Last edit: by Jamal. |
VO 32 Bit limitations 14 Jul 2018 12:58 #5480
|
Very nice, something what I I was already looking for. Daniel, the author of the tool Wolfgang had found, has confirmed that he makes the same as the Visual Studio tool. He also said that on 32-Bit Windows computers the flags will be ignored and do not disturb. Arne |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 20 Jul 2018 14:40 #5564
|
We are making now some beta tests in our company. It looks good until now. I have tested that it works also with XSharp 32-bit programs. Although the memory management of Dot.net is better I have seen that also in Dotnet we reach quickly the 2 GB limit. And the move from 32 bit to 64 bit is another big challenge after having moved from VO to XSharp. Arne |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 20 Jul 2018 18:17 #5565
|
The first problematic computer: The program starts shortly with the following message: Processor stack fault FFF1FF6C Ok It seems to be the standard VO error message window. On my computer the same EXE file works fine. The other machine is nearly identical (same motherboard, same Windows 10 64Bit) Does anybody has an idea where this can come from? Arne |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 21 Jul 2018 02:22 #5567
|
Hi Arne, this is the list: 1. firmware update on motherboard BIOS. Intel? AMD? 2. chipsets updates. 1803 updates really wreak a lot of computers with outdated firmware. Just a thought. BTW, thanks for starting this thread. I copied all the code Jamal have shared and all thoughts herein: Wolfgang, Robert and yours. Now, my apps is "LAA optimized"..... Regards, Rene -- Rene Pajaron |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 22 Jul 2018 04:03 #5570
|
Does this help? From: harbour.github.io/the-oasis/clipper-5.html#ss5.6 5.6 What is an "Unrecoverable Error 650: Processor Stack Fault"? It's a result of infinite recursion, which occurs when a function calls itself, which calls itself, which calls itself, and so on without stopping. Most often this occurs when your error handler has an error, which triggers the error handler
|
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 23 Jul 2018 14:19 #5591
|
It is strange. Until now two computers have been found, where the program does not start anymore. - The error happens in the EXE before the first self written line of code - It happens in Vo28run!_ConvertResult - this calls vo28run!_StackErr and then produces the error message on the screen - It happens exactly with a simple console application. Arne |
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 23 Jul 2018 16:24 #5593
|
Can you provide the code for simple console app? How did you apply the LMA feature? Jamal
|
Please Log in or Create an account to join the conversation. |
VO 32 Bit limitations 23 Jul 2018 16:39 #5594
|
Hi Jamal, >Can you provide the code for simple console app? I used the wizzard in VO under Basic/Console App and created the exe. > How did you apply the LMA feature? I use the editbin program from VS editbin.exe /largeaddressaware %1 Until now 2 out of about 15 tested computers show this behavior, on the other computers the program starts as desired. Arne |
Please Log in or Create an account to join the conversation. |
|