• Hi All

    Please note that at the Chandoo.org Forums there is Zero Tolerance to Spam

    Post Spam and you Will Be Deleted as a User

    Hui...

  • When starting a new post, to receive a quicker and more targeted answer, Please include a sample file in the initial post.

insufficient memory

Bossie

New Member
I have a sourcefile (xlsm) with 18000 rows of data.

I need to copy two cells from each line to different sheets on different files (one per person).

So I browsed the internet and found a module that does the trick:

· It asks in which folder the target files are

· In a loop

o it opens file by file

o in a loop it opens sheet by sheet

o it finds the rows in the sourcefile with the data to copy

o it copies the data to the sheet

o it closes and saves the file

This works fine fora bout 10 files and then it shows an error ‘insufficient memory’

I need to make this work for about 1000 files so this is a problem.


Any thoughts on how to avoid this message?

this is the code (messy, I know)

Code:
Sub teamscanteam()

'initialisaties

Dim wb As Workbook
Dim myPath As String
Dim myFile As String
Dim myExtension As String
Dim FldrPicker As FileDialog
Dim Current As Worksheet
Dim persnr As String
Dim bronbestand As Workbook
Dim bronsht As Worksheet
Dim productpersnr As String
Dim gevonden
Dim gevondenlyn

'Optimimaliseren snelheid uitvoering macro
'  Application.ScreenUpdating = False
'  Application.EnableEvents = False
'  Application.Calculation = xlCalculationManual

bronpad = "\\antwerpen.local\Doc\PM\1_16_Monitoren\05_Terugkerende_rapportering\22_teamscan\"
Set bronbestand = Workbooks("teamscan3_zonderformules3.xlsm")
Set bronsht = bronbestand.Worksheets("koppelingenSB")

'bepaal het pad waar de bestanden staan
Set FldrPicker = Application.FileDialog(msoFileDialogFolderPicker)
    With FldrPicker
      .Title = "Select A Target Folder"
      .AllowMultiSelect = False
        If .Show <> -1 Then GoTo NextCode
        myPath = .SelectedItems(1) & "\"
    End With

'In Case of Cancel
NextCode:
  myPath = myPath
  If myPath = "" Then GoTo ResetSettings

'extensie van de bestanden bepalen
 myExtension = "*.xls*"

'Target Path with Ending Extention
  myFile = Dir(myPath & myExtension)

'Loop door alle excelbestanden in de directory
Do While myFile <> ""
   
    'Set variable equal to opened workbook
    Set wb = Workbooks.Open(filename:=myPath & myFile)
       
    'Ensure Workbook has opened before moving on to next line of code
    DoEvents
       
    ' Loop door alle sheets in het geopende workbook
    For Each Current In Worksheets
   
        ' Haal het personeelsnummer op
        persnr = CStr(Current.Range("B3").Value)
             
        'testers
        Debug.Print Current.Name
        Debug.Print persnr
       
        For productlyn = 1 To 18107
           productpersnr = bronsht.Cells(productlyn, 1).Value
           If productpersnr = persnr Then gevondenlyn = productlyn
        Next productlyn
   
    Next
       
    'Save and Close Workbook
    wb.Close SaveChanges:=True
         
    'Ensure Workbook has closed before moving on to next line of code
    DoEvents
   
    'Get next file name
    myFile = Dir

Loop

'Message Box when tasks are completed
  MsgBox "Task Complete!"

ResetSettings:
  'Reset Macro Optimization Settings
'    Application.EnableEvents = True
'    Application.Calculation = xlCalculationAutomatic
'    Application.ScreenUpdating = True

End Sub
 
Hi ,

I don't think this will make a difference , but try anyway.

Just before the Loop statement in your code , include the following line of code :

Set wb = Nothing

Narayan
 
thx Narayan!

I did that and reduced the sourcefile by deleting the columns I don't need.
Now the module works (at least for the 10 files I tested it with)
But now when I quit excel it says "Excel has stopped working') so I suspect it still struggles with memory...

I'm working with excel 2010, 64bit on a hp zbook (core I7)...

Any thoughts?


(below is the full module)

Code:
Sub teamscanteam()

'initialisaties

Dim wb As Workbook
Dim myPath As String
Dim myFile As String
Dim myExtension As String
Dim FldrPicker As FileDialog
Dim Current As Worksheet
Dim persnr As String
Dim bronbestand As Workbook
Dim bronsht As Worksheet
Dim productpersnr As String
Dim gevonden
Dim gevondenlyn
Dim productpercent As String
Dim producttekst As String
Dim doellyn As Integer


'Optimimaliseren snelheid uitvoering macro
'  Application.ScreenUpdating = False
'  Application.EnableEvents = False
'  Application.Calculation = xlCalculationManual

bronpad = "\\antwerpen.local\Doc\PM\1_16_Monitoren\05_Terugkerende_rapportering\22_teamscan\"
Set bronbestand = Workbooks("prodkoppelSB.xlsx")
Set bronsht = bronbestand.Worksheets("koppelingenSB")
'bepaal het pad waar de bestanden staan
Set FldrPicker = Application.FileDialog(msoFileDialogFolderPicker)
    With FldrPicker
      .Title = "Select A Target Folder"
      .AllowMultiSelect = False
        If .Show <> -1 Then GoTo NextCode
        myPath = .SelectedItems(1) & "\"
    End With

'In Case of Cancel
NextCode:
  myPath = myPath
  If myPath = "" Then GoTo ResetSettings

'extensie van de bestanden bepalen
 myExtension = "*.xls*"

'Target Path with Ending Extention
  myFile = Dir(myPath & myExtension)

'Loop door alle excelbestanden in de directory
Do While myFile <> ""
   
    'Set variable equal to opened workbook
    Set wb = Workbooks.Open(filename:=myPath & myFile)
       
    'Ensure Workbook has opened before moving on to next line of code
    DoEvents
       
    ' Loop door alle sheets in het geopende workbook
    For Each Current In Worksheets
   
        ' Haal het personeelsnummer op
        persnr = CStr(Current.Range("B3").Value)
        doellyn = 18
        'testers
       ' Debug.Print Current.Name
       ' Debug.Print persnr
       
        For productlyn = 1 To 18107
           productpersnr = bronsht.Cells(productlyn, 1).Value
           If productpersnr = persnr Then gevonden = 1 Else gevonden = 0
            If gevonden = 1 Then
             productpercent = bronsht.Cells(productlyn, 2).Value
             producttekst = bronsht.Cells(productlyn, 3).Value
             Current.Cells(doellyn, 2).Value = productpercent
             Current.Cells(doellyn, 3).Value = producttekst
             doellyn = doellyn + 1
            End If
        Next productlyn
        For wisser = doellyn To 34
        Current.Cells(doellyn, 2).Value = ""
        Current.Cells(doellyn, 3).Value = ""
        Next wisser
    Next
       
    'Save and Close Workbook
    wb.Close SaveChanges:=True
         
    'Ensure Workbook has closed before moving on to next line of code
    DoEvents
   
    'Get next file name
    myFile = Dir
Set wb = Nothing

Loop

'Message Box when tasks are completed
  MsgBox "Task Complete!"

ResetSettings:
  'Reset Macro Optimization Settings
'    Application.EnableEvents = True
'    Application.Calculation = xlCalculationAutomatic
'    Application.ScreenUpdating = True

End Sub
 
Hi ,

Can you indicate how much of system RAM is available on your computer ?

Since you have 64-bit Excel , it can handle all the memory you have ; it might very well be your hardware which is the limitation.

Narayan
 
Hi,

it's 8Gb. If it's a hardwareproblem, then it's unsolvable...(my employer is cutting costs, so extra investments (no matter how small) will take a few months to get cleared...

I was thinking: could it be possible to clear memory while executing the module so it can be re-used? Could it be that every loop causes a 'residu' in memory?


Hugo
 
The code looks ok, without trying it

I have done similar things on PC's with minimal memory and have had no problems

What line is it on when it crashes ?
Does it always stop at the same spot ?

I would put a few debug print lines inside the loops to show how many lines you are reading, files open etc, size of arrays

Can you post a sample file here so we can try the code?
 
is 18107 the number of rows?
Can it be read from the file for each Worrksheet?
 
Hi Hui,
thanks for joining this thread.

I uploaded the files (different names but the content is the same: the source and the target).
The module (when I tested it with 10 targetfiles) didn't crash until the end.
18107 is indeed the number of rows in my sourcefile.
grtz

Hugo
 

Attachments

Hi ,

If you have 8 GB RAM , I think the hardware should not be the limitation.

To test this , why don't we split up the code into 2 sections , one which does the file open and close , and the other which goes through each workbook.

Try the following code and see whether it runs to completion ; try it with the folder which has 1000 files.
Code:
Option Explicit

Sub teamscanteam()
'  initialisaties

    Dim FldrPicker As FileDialog
    Dim bronpad As String, myPath As String, myFile As String, myExtension As String, persnr As String, productpersnr As String
    Dim bronbestand As Workbook, wb As Workbook
    Dim bronsht As Worksheet, Current As Worksheet
    Dim gevonden As Variant, gevondenlyn As Variant
    Dim productlyn As Integer
   
'  Optimimaliseren snelheid uitvoering macro
'  Application.ScreenUpdating = False
'  Application.EnableEvents = False
'  Application.Calculation = xlCalculationManual

    bronpad = "\\antwerpen.local\Doc\PM\1_16_Monitoren\05_Terugkerende_rapportering\22_teamscan\"
    Set bronbestand = Workbooks("teamscan3_zonderformules3.xlsm")
    Set bronsht = bronbestand.Worksheets("koppelingenSB")

'  bepaal het pad waar de bestanden staan
    Set FldrPicker = Application.FileDialog(msoFileDialogFolderPicker)
    With FldrPicker
        .Title = "Select A Target Folder"
        .AllowMultiSelect = False
        If .Show <> -1 Then GoTo NextCode
        myPath = .SelectedItems(1) & "\"
    End With

'  In Case of Cancel
NextCode:
    myPath = myPath
    If myPath = "" Then GoTo ResetSettings

'  extensie van de bestanden bepalen
    myExtension = "*.xls*"

'  Target Path with Ending Extention
    myFile = Dir(myPath & myExtension)

'  Loop door alle excelbestanden in de directory
    Do While myFile <> ""
'      Set variable equal to opened workbook
      Set wb = Workbooks.Open(Filename:=myPath & myFile)
     
'      Ensure Workbook has opened before moving on to next line of code
      DoEvents
      GoTo SkipLoop:
'      Loop door alle sheets in het geopende workbook
      For Each Current In Worksheets
'          Haal het personeelsnummer op
          persnr = CStr(Current.Range("B3").Value)
           
'          testers
          Debug.Print Current.Name
          Debug.Print persnr
     
          For productlyn = 1 To 18107
              productpersnr = bronsht.Cells(productlyn, 1).Value
              If productpersnr = persnr Then gevondenlyn = productlyn
          Next productlyn
      Next
SkipLoop:
'      Save and Close Workbook
      wb.Close SaveChanges:=True
       
'      Ensure Workbook has closed before moving on to next line of code
      DoEvents
 
'      Get next file name
      myFile = Dir
    Loop

'  Message Box when tasks are completed
    MsgBox "Task Complete!"

ResetSettings:
'  Reset Macro Optimization Settings
'  Application.EnableEvents = True
'  Application.Calculation = xlCalculationAutomatic
'  Application.ScreenUpdating = True
End Sub
Just BTW , what is the size of your biggest workbook ?

Narayan
 
Hi Narayan,

This part works without problems. Can the part between the 'skip loop' lines be the cause of the problems then?

(filesize: I uploaded the source file and one of the target files, the target files are all the same size (approximatly), the sourcefile I reduced to 543Kb

thx

Hugo
 
I 'undid' the skip loop part (by adding quotes) and with the reduced sourcefile and with a few 100 targetfiles, it works without errors.
(I had to add a line: doellyn=18 by the start of every loop through the targetfiles to get it working but that was easy :-) )
 
Back
Top