[SOLVED] memory problem

fanderal · 2022-12-15 19:32:39

Layout
sda: SSD, Ceres, Fluxbox desktop, repo only pkgs, 3 partitions
sdb: SSD, same as above
sdc: HDD, storage, 3 partitions + empty extended partition
sdd: HDD, storage, 1 partition

Sda Ceres was installed maybe 4 years ago, currently using a 4.19.x kernel and normally boots to the desktop using ~105MB ram. Sdb Ceres was installed near 2 years ago, uses a 6.0.0-5-amd64 kernel and normally boots to the desktop using ~130MB ram. Sda now uses ~330MB ram and sdb now uses ~385MB ram.

Last week, same time the ram jumped up, there was also a UUID problem. Eg: when in sdb and clicking the Fluxbox menu to mount/open sdc2, sda1 would mount/open. Found a ton of debug msgs in the logs and after searching, disabled os-prober in sda and sdb. Grub is installed on sda and installing grub-customizer seems to have solved the UUID problem.

For the hardware, filesystem checks at boot all show clean, and fsck, badblocks and smartctl show no problems on sda or sdb. Log searches for 'fail' found nothing but "snd_hda_intel: probe failed with error -2" on sda/sdb yet sound works fine. Mobo has two 2GB ram sticks and booting each stick separately showed only the same high memory use.

At the sdb desktop with only a root terminal open:

# free -m
               total        used        free      shared  buff/cache   available
Mem:            3771         463        2734          55         845        3308
Swap:           1824           0        1824

# inxi -tm
Processes:
  System RAM: total: 3.68 GiB used: 490.2 MiB (13.0%)
  Memory top: 5 ( 1 processes) of 119
  1: mem: 98.7 MiB (2.6%) command: xorg pid: 2280
  2: mem: 20.1 MiB (0.5%) command: conky pid: 2290
  3: mem: 13.9 MiB (0.3%) command: xterm pid: 11043
  4: mem: 11.2 MiB (0.2%) command: fluxbox pid: 2289
  5: mem: 8.57 MiB (0.2%) command: at-spi-bus-launcher pid: 831

In the list from the first cmd below, spot checks of running processes show nothing obvious, seem normal and are consistent between sda and sdb.

$ ps -eo pid,class,stat,vsz,rss,comm
# cat /proc/PID/smaps | grep -i rss |  awk '{Total+=$2} END {print Total/1024" MB"}'
# smem -k | sed -e '1p' -e '/conky/!d' | grep -v sed

Recently: on Dec 1, 46 pkgs were updated. On Dec 3, I did a semi-annual cleanup using debfoster and deleted ~30 pkgs. Ran localepurge same day. On Dec 6, 70 pkgs were updated. I noticed the UUID and ram problems in sdb first and then sda on Dec 9. Thought it might involve the updated VirtualBox-7.0.4-dfsg-4 on sdb, but it's not installed on sda. Can't resolve this until I can find what's using the excess memory.

Appreciate any suggestions.

Head_on_a_Stick · 2022-12-15 19:55:53

To check memory usage try https://github.com/pixelb/ps_mem.

EDIT: also:

awk '/Rss/{Total+=$2} END {print Total/1024" MiB"}' /proc/*/smaps

Useless use of cat and useless use of grep. We don't need no stinkin' pipes...

Last edited by Head_on_a_Stick (2022-12-15 21:07:13)

aluma · 2022-12-15 20:50:24

In general, that is the

top

command.
In order for it to sort processes by memory usage, you must press ">".

fanderal · 2022-12-15 21:50:04

Thanks for the thought, HoaS. I still have/use the ps-mem.py.sh script from 2015. I installed the one you linked and below are the results with just a root terminal on the desktop.

Old

# ps-mem.py.sh

 Private + Shared = RAM used    Program 

  4.0 KiB +  93.0 KiB =  97.0 KiB       startpar
 36.0 KiB +  94.0 KiB = 130.0 KiB       bootlogd
 76.0 KiB + 180.0 KiB = 256.0 KiB       init
100.0 KiB + 180.0 KiB = 280.0 KiB       atd
240.0 KiB + 371.0 KiB = 611.0 KiB       cron
508.0 KiB + 317.0 KiB = 825.0 KiB       dhclient
264.0 KiB + 572.0 KiB = 836.0 KiB       xinit
528.0 KiB + 459.0 KiB = 987.0 KiB       udevd
504.0 KiB + 548.0 KiB =   1.0 MiB       dbus-daemon
496.0 KiB + 902.0 KiB =   1.4 MiB       login
964.0 KiB + 808.0 KiB =   1.7 MiB       rsyslogd
  1.0 MiB + 878.0 KiB =   1.9 MiB       su
688.0 KiB +   1.2 MiB =   1.9 MiB       getty (5)
  1.3 MiB + 568.0 KiB =   1.9 MiB       elogind-daemon
  3.8 MiB +   4.3 MiB =   8.1 MiB       fluxbox
  4.2 MiB +   5.9 MiB =  10.0 MiB       bash (3)
  8.3 MiB +   7.1 MiB =  15.4 MiB       xterm
 10.8 MiB +   9.4 MiB =  20.2 MiB       conky
101.1 MiB +  48.9 MiB = 150.1 MiB       Xorg
---------------------------------
                        217.5 MiB
=================================

New

# ps_mem -d

 Private  +   Shared  =  RAM used       Program[pid]

  4.0 KiB +  70.5 KiB =  74.5 KiB       startpar [671]
 36.0 KiB +  44.5 KiB =  80.5 KiB       bootlogd [670]
100.0 KiB +  45.5 KiB = 145.5 KiB       atd [1400]
 76.0 KiB + 108.5 KiB = 184.5 KiB       init [1]
136.0 KiB +  84.5 KiB = 220.5 KiB       getty [1465]
136.0 KiB +  84.5 KiB = 220.5 KiB       getty [1468]
144.0 KiB +  83.5 KiB = 227.5 KiB       getty [1466]
136.0 KiB +  93.5 KiB = 229.5 KiB       getty [1467]
136.0 KiB +  99.5 KiB = 235.5 KiB       getty [1464]
240.0 KiB + 116.5 KiB = 356.5 KiB       cron [1437]
264.0 KiB + 280.5 KiB = 544.5 KiB       xinit [26536]
528.0 KiB +  50.5 KiB = 578.5 KiB       udevd [412]
508.0 KiB +  84.5 KiB = 592.5 KiB       dhclient [1114]
504.0 KiB + 185.5 KiB = 689.5 KiB       dbus-daemon [1439]
496.0 KiB + 268.5 KiB = 764.5 KiB       login [1463]
964.0 KiB +  69.5 KiB =   1.0 MiB       rsyslogd [1374]
  1.0 MiB + 344.5 KiB =   1.3 MiB       su [26612]
  1.3 MiB + 114.5 KiB =   1.5 MiB       elogind-daemon [1444]
  1.4 MiB + 500.5 KiB =   1.9 MiB       bash [1473]
  1.4 MiB + 596.5 KiB =   1.9 MiB       bash [26653]
  1.4 MiB + 592.5 KiB =   2.0 MiB       bash [26590]
  3.8 MiB +   1.7 MiB =   5.5 MiB       fluxbox [26547]
  8.3 MiB +   1.2 MiB =   9.5 MiB       xterm [26588]
 13.3 MiB +   1.6 MiB =  14.9 MiB       conky [26548]
101.0 MiB +   1.3 MiB = 102.2 MiB       Xorg [26537]
---------------------------------
                        146.7 MiB
=================================

The old and new scripts show different memory and shared memory values, and a 70MB difference in total. Some spot checks of xorg, conky and free:

xorg

# smem -k | sed -e '1p' -e '/xorg/!d' | grep -v sed
  PID User     Command                         Swap      USS      PSS      RSS 
26537 (user)    /usr/lib/xorg/Xorg -noliste        0    99.8M   102.8M   109.9M 

# cat /proc/26537/smaps | grep -i rss |  awk '{Total+=$2} END {print Total/1024" MB"}'
112.652 MB

conky

#  smem -k | sed -e '1p' -e '/conky/!d' | grep -v sed
  PID User     Command                         Swap      USS      PSS      RSS 
26548 (user)    conky --pause=1 -c /home/xe        0    13.0M    14.2M    20.1M 

#  cat /proc/26548/smaps | grep -i rss |  awk '{Total+=$2} END {print Total/1024" MB"}'
20.1133 MB

# free -m
               total        used        free      shared  buff/cache   available
Mem:            3771         455        2762          52         835        3316
Swap:           1824           4        1820

I'm probably missing something but how do you reconcile the differences between the totals of either ps_mem with free?

fanderal · 2022-12-15 22:42:42

Thanks, aluma, top after pressing ">"

Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.3 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 3771.9 total, 2759.9 free, 451.8 used, 848.0 buff/cache
MiB Swap: 1825.0 total, 1820.5 free, 4.5 used. 3320.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26537 (usr) 20 0 615276 109048 60796 S 0.7 2.8 1:57.31 Xorg
26548 (usr) 20 0 625872 20612 14220 S 1.3 0.5 1:41.38 conky
17393 (usr) 20 0 23512 13992 8072 S 0.0 0.4 0:00.17 xterm
26547 (usr) 20 0 21848 11672 8652 S 0.0 0.3 0:06.10 fluxbox
4536 (usr) 20 0 161004 10088 5444 S 0.0 0.3 0:01.21 at-spi2-registr
4511 (usr) 20 0 308332 8628 5988 S 0.0 0.2 0:00.00 at-spi-bus-laun
17395 (usr) 20 0 7984 4936 3516 S 0.0 0.1 0:00.01 bash
1473 (usr) 20 0 8104 4784 3324 S 0.0 0.1 0:00.02 bash
21935 (usr) 20 0 8172 4668 2664 R 0.0 0.1 0:00.01 top
4522 (usr) 20 0 156308 4432 4084 S 0.0 0.1 0:00.00 dconf-service
4516 (usr) 20 0 4308 3068 2820 S 0.0 0.1 0:00.41 dbus-daemon
1444 root 20 0 38084 3032 2656 S 0.0 0.1 0:00.01 elogind-daemon
1463 root 20 0 4868 2336 1856 S 0.0 0.1 0:00.02 login
1114 root 20 0 5868 2172 1960 S 0.0 0.1 0:00.00 dhclient
1437 root 20 0 6608 2140 1932 S 0.0 0.1 0:00.09 cron
4508 (usr) 20 0 6672 2004 1664 S 0.0 0.1 0:00.00 dbus-launch
4509 (usr) 20 0 4408 1928 1640 S 0.0 0.0 0:00.00 dbus-daemon
1439 message+ 20 0 4436 1832 1580 S 0.0 0.0 0:00.13 dbus-daemon
1 root 20 0 3296 1820 1776 S 0.0 0.0 0:01.30 init
1374 root 20 0 218332 1712 1304 S 0.0 0.0 0:00.01 rsyslogd
412 root 20 0 22892 1680 1344 S 0.0 0.0 0:00.15 udevd
26536 (usr) 20 0 6176 1324 1152 S 0.0 0.0 0:00.00 xinit
671 root 20 0 2484 900 900 S 0.0 0.0 0:00.00 startpar
1467 root 20 0 5872 860 772 S 0.0 0.0 0:00.00 getty
1468 root 20 0 5872 860 776 S 0.0 0.0 0:00.00 getty
1465 root 20 0 5872 796 708 S 0.0 0.0 0:00.00 getty
1466 root 20 0 5872 780 692 S 0.0 0.0 0:00.00 getty
1464 root 20 0 5872 760 672 S 0.0 0.0 0:00.00 getty
1400 daemon 20 0 3580 84 0 S 0.0 0.0 0:00.00 atd
670 root 20 0 2564 20 0 S 0.0 0.0 0:06.12 bootlogd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd

Head_on_a_Stick · 2022-12-15 22:44:06

fanderal wrote:

how do you reconcile the differences between the totals of either ps_mem with free?

Shared memory usage in Linux is complicated, check the ps_mem code comments for details.

Looks like X is bloated for some reason. Can we see the log?

Last edited by Head_on_a_Stick (2022-12-15 23:10:42)

fanderal · 2022-12-15 23:40:19

Head_on_a_Stick wrote:

Shared memory usage in Linux is complicated

Ain't that the truth.

Most recent Xorg log, Dec 14
https://droptext.cc/napw

FWIW, my Dell 24" bit the dust about five months ago and currently using an old Motorola 37".

inxi -G
Graphics:
  Device-1: Intel 4th Generation Core Processor Family Integrated
    Graphics driver: i915 v: kernel
  Display: server: X.Org v: 1.21.1.4 driver: X:
    loaded: modesetting dri: crocus gpu: i915
    resolution: 1920x1080~60Hz
  API: OpenGL v: 4.6 Mesa 22.3.0 renderer: Mesa Intel HD Graphics
    4400 (HSW GT2)

Head_on_a_Stick · 2022-12-16 10:46:01

The log looks normal. Anything in ~/.xsession-errors?

X does tend to bloat over time, or at least it did before I dropped it. Have you considered Wayland instead?

ralph.ronnquist · 2022-12-16 12:07:42

@fanderal, there may be some ram used by various tmpfs mounts, which use RAM but are not reported together with process RAM usage. You could check with:

df -h | grep -E 'Filesystem|tmpfs'

fanderal · 2022-12-16 20:17:42

@Head_on_a_Stick

Thanks for taking a look. The ~/.xsession-errors file's dated Dec 2021, has a little over one page of lines and never updated. Some lines:

dbus-update-activation-environment: systemd --user not found, ignoring --systemd argument
dbus-update-activation-environment: setting NODM_X_TIMEOUT=300
dbus-update-activation-environment: setting NODM_FIRST_VT=7
dbus-update-activation-environment: setting MAIL=/var/mail/root
dbus-update-activation-environment: setting USER=(usr)
dbus-update-activation-environment: setting HOME=/home/(usr)
dbus-update-activation-environment: setting GTK_MODULES=gail:atk-bridge
dbus-update-activation-environment: setting DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-FuMn6Eboxq,guid=9e57b5f520e2f6a60596091661b6297e
dbus-update-activation-environment: setting QT_QPA_PLATFORMTHEME=qt5ct
dbus-update-activation-environment: setting TERM=linux
dbus-update-activation-environment: setting WINDOWPATH=7
dbus-update-activation-environment: setting DISPLAY=:0
dbus-update-activation-environment: setting LANG=en_US.UTF-8
dbus-update-activation-environment: setting SHELL=/bin/bash
dbus-update-activation-environment: setting QT_ACCESSIBILITY=1

Have you considered Wayland

No I haven't, although I've read some of your posts about it. Perhaps it's time to give it a spin and see for myself.

fanderal · 2022-12-16 20:20:03

@ralph.ronnquist

Thanks for the suggestion.

$ df -h | grep -E 'Filesystem|tmpfs'
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           378M  748K  377M   1% /run
tmpfs           5.0M  8.0K  5.0M   1% /run/lock
tmpfs           1.1G     0  1.1G   0% /dev/shm
tmpfs           378M  4.0K  378M   1% /run/user/1000

Been searching for an explanation of the above results but haven't found anything useful, or what it may have to do with suddenly showing excessive memory use. Appreciate any help.

Head_on_a_Stick · 2022-12-16 21:01:20

Did the memory usage increase before the xserver-xorg-core 21.1.5 release? Perhaps there is a regression.

How are you starting fluxbox?

fanderal · 2022-12-17 01:33:01

The last xserver-xorg-core update was Nov 15, 2:21.1.4-2 -> 2:21.1.4-3. Since your last post, I updated and found the 21.1.5-1 version included with 115 available updates. Installed the updates and rebooted to get a fresh read. Same memory problem using 21.1.4-3 until earlier today, and 21.1.5-1 now.

It's a console login and 'xinit' starts the fluxbox desktop. The .xinitrc file has 'exec fluxbox,' feh loads the background, starts conky and xnumlock. Maybe 2/3 years ago, I had to create an '.xserverrc' file for xinit to start the desktop.

Head_on_a_Stick · 2022-12-17 11:54:27

fanderal wrote:

The .xinitrc file has 'exec fluxbox,' feh loads the background, starts conky and xnumlock.

I would prefer to see the actual file contents, if you don't mind. TIA.

fanderal wrote:

Maybe 2/3 years ago, I had to create an '.xserverrc' file for xinit to start the desktop.

So what happens if you move that somewhere else and run

startx /usr/bin/fluxbox

That won't give you a full desktop but it will help eliminate custom startup scripts as a culprit here.

fanderal · 2022-12-17 19:06:39

.xinitrc

#!/bin/sh
#
numlockx &
conky --pause=1 -c ~/.conky/conky.conf &
# feh --bg-scale '~/.fluxbox/backgrounds/neon-surf.jpg' &
# feh --bg-scale '~/.fluxbox/backgrounds/la-playa.jpg' &
feh --bg-scale '~/.fluxbox/backgrounds/night-poppies-1.jpg' &
#
exec fluxbox

The line for feh invokes:

.fehbg

#!/bin/sh
feh --no-fehbg --bg-scale '~/.fluxbox/backgrounds/night-poppies-1.jpg'

And FWIW:

.xserverrc

#!/bin/sh
exec /usr/bin/Xorg -nolisten tcp "$@" vt$XDG_VTNR

Head_on_a_Stick wrote:

So what happens if you move .xserverrc somewhere else and run startx /usr/bin/fluxbox

Got the Fluxbox desktop, minus the background, etc.

Appreciate your thoughts, and I'm open to whatever info you ask for. However, just so you know, a little over halfway down the second page of 'Show your desktop (rebooted)' is my desktop from Sep 2018, #42. The .xinitrc and .fehbg files have been edited maybe two or three times since then for the background, and the .xsession file was never edited after creating it 2+ years ago.

BUT... what you asked gave me an idea. The only changes on these OSs, aside from updates and infrequent color scheme edits, are occasional downloads of videos, etc. The excess memory might be from a download, a dot file edit or something else. I moved everything except .Xauthority, .xinitrc (commented out conky, feh, xnumlock) and .xserver to 'old.home.stuff' and rebooted. Same excessive memory.

Head_on_a_Stick · 2022-12-17 19:45:17

Which graphics chip are you running? If it's Intel then remove xserver-xorg-video-intel. That driver hasn't been properly maintained for about 10 years and it's buggy as hell. X's built-in modesetting DDX driver should offer a better experience if you don't have old hardware. It even has a new TearFree option to help combat X's completely broken compositing model, which is nice.

Just for the record here are my values with Xorg 21.5.1 with a Wayland comparison (amdgpu, 1920x1080):

$ doas ps_mem | grep 'Xorg\|openbox\|sway$'
  7.4 MiB +   1.4 MiB =   8.8 MiB	openbox
 36.3 MiB +  17.0 MiB =  53.3 MiB	sway
 51.1 MiB +  32.7 MiB =  83.9 MiB	Xorg
$

EDIT: I use Arch btw

EDIT2: changed awk to grep just in case andyprough takes the piss.

EDIT3: s/composting/compositing/. Possibly more accurate with the typo though...

Last edited by Head_on_a_Stick (2022-12-17 20:01:14)

fanderal · 2022-12-17 22:39:59

The xserver-xorg-video-intel pkg isn't installed, nor anything intel (eg: non-free microcode, driver, shaders) except libdrm-intel1, as a dependency of libgl1-mesa-dri. X is using mesa's OpenGL stuff... video works great online or local. The HD4400 graphics chip is integrated with the Core i3-4160 CPU.

$ inxi -G
Graphics:
  Device-1: Intel 4th Generation Core Processor Family Integrated
    Graphics driver: i915 v: kernel
  Display: server: X.Org v: 1.21.1.5 driver: X:
    loaded: modesetting dri: crocus gpu: i915
    resolution: 1920x1080~60Hz
  API: OpenGL v: 4.6 Mesa 22.3.1 renderer: Mesa Intel HD Graphics
    4400 (HSW GT2)

Don't have/use a compositor, just fluxbox's pseudo transparency. With Xorg 21.5.1:

doas ps_mem | grep 'Xorg\|fluxbox'
  4.8 MiB + 625.5 KiB =   5.4 MiB       fluxbox
 52.6 MiB +  27.3 MiB =  79.8 MiB       Xorg

Thanks for doas, didn't know of it until your post.

Head_on_a_Stick · 2022-12-17 22:49:36

So that memory use looks normal. Is free still showing an excess? The procps package hasn't been updated since April so I don't think it's the free command itself that's changed.

Anyway I'm out of ideas here, sorry.

And if you have a Haswell CPU you need the Intel microcode package to stop it crashing randomly.

andyprough · 2022-12-18 00:23:45

Head_on_a_Stick wrote:

EDIT2: changed awk to grep just in case andyprough takes the piss.

In theological studies that's called "accommodation" - God spoke to humans in baby talk because they couldn't understand His mind or thoughts. HOAS speaks to the intellectually challenged folks like me in baby grep because we can't understand the awk/sed mind or awk/sed thoughts.

@fanderal - I'm assuming you have been keeping your page memory and dentries and inodes nice and clean? As root:

sync; echo 1 > /proc/sys/vm/drop_caches
sync; echo 2 > /proc/sys/vm/drop_caches

Last edited by andyprough (2022-12-18 00:32:07)

fanderal · 2022-12-18 02:21:00

@Head_on_a_Stick

Glad the doas in my previous post looked normal. I just ran it again. Any idea why xorg is almost double what it was with the same files open on the desktop?

doas ps_mem | grep 'Xorg\|fluxbox'
  4.2 MiB +   1.1 MiB =   5.3 MiB       fluxbox
100.4 MiB +   4.7 MiB = 105.0 MiB       Xorg

Yes, free still shows excess memory. With just xterm on the desktop:

free -m
               total        used        free      shared  buff/cache   available
Mem:            3771         465        2939          53         630        3306
Swap:           1824           0        1824

Procps was updated on Dec 6:
[UPGRADE] procps:amd64 2:3.3.17-7.1devuan1 -> 2:4.0.2-1devuan1
along with this:
[REMOVE, NOT USED] libprocps8:amd64 2:3.3.17-7.1devuan1
The libprocps8 pkg is in the Debian repos but not in Devuan's... haven't looked into why.
Was updated again on Dec 16:
[UPGRADE] procps:amd64 2:4.0.2-1devuan1 -> 2:4.0.2-2devuan1

Yes, the CPU's a Haswell but the intel-microcode and iucode-tool pkgs are not installed. The firmware-linux and firmware-linux-nonfree pkgs are not installed, either. Neither of the sda/sdb OSs have had any of those pkgs installed for well over a year. The kernel, though, has about ten built-in Intel modules for sound, better function and security.

I'm out of ideas here, sorry.

No worries, my thanks for all you've done to help.

fanderal · 2022-12-18 02:24:48

@andyprough

been keeping your page memory and dentries and inodes nice and clean?

Yes. I don't usually boot back and forth between sda/sdb as I've done this past week. I'm normally in one or the other for a month or more without a reboot. That sync cmd is in the fluxbox menu and I run it a couple of times a week. Appreciate the thought.

andyprough · 2022-12-18 03:35:54

Probably systemd crawled off of one of HOAS's systems up into the forum software and then down into your computer and started bloating the place up, putting up new wallpaper, buying expensive new furniture, running up huge credit card bills...

fanderal · 2022-12-18 06:15:14

... and coming up with a miraculous fix for a 'modest' fee.

aluma · 2022-12-18 08:48:16

@fanderal
You can open /var/log/Xorg.0.log and see what it downloads.

Head_on_a_Stick · 2022-12-18 11:49:07

andyprough wrote:

sync; echo 1 > /proc/sys/vm/drop_caches
sync; echo 2 > /proc/sys/vm/drop_caches

The kernel is quite capable of dropping the cache by itself if the memory is needed. And anyway the free output clearly shows that the buffers are almost empty. See also https://www.linuxatemyram.com/.

aluma wrote:

/var/log/Xorg.0.log

That location is only used if X is running under root. I think the OP's log will be under ~/.local/share/xorg/ but their use of custom startup scripts complicates the picture somewhat.

The officially official Devuan Forum!

#1 2022-12-15 19:32:39

[SOLVED] memory problem

#2 2022-12-15 19:55:53

Re: [SOLVED] memory problem

#3 2022-12-15 20:50:24

Re: [SOLVED] memory problem

#4 2022-12-15 21:50:04

Re: [SOLVED] memory problem

#5 2022-12-15 22:42:42

Re: [SOLVED] memory problem

#6 2022-12-15 22:44:06

Re: [SOLVED] memory problem

#7 2022-12-15 23:40:19

Re: [SOLVED] memory problem

#8 2022-12-16 10:46:01

Re: [SOLVED] memory problem

#9 2022-12-16 12:07:42

Re: [SOLVED] memory problem

#10 2022-12-16 20:17:42

Re: [SOLVED] memory problem

#11 2022-12-16 20:20:03

Re: [SOLVED] memory problem

#12 2022-12-16 21:01:20

Re: [SOLVED] memory problem

#13 2022-12-17 01:33:01

Re: [SOLVED] memory problem

#14 2022-12-17 11:54:27

Re: [SOLVED] memory problem

#15 2022-12-17 19:06:39

Re: [SOLVED] memory problem

#16 2022-12-17 19:45:17

Re: [SOLVED] memory problem

#17 2022-12-17 22:39:59

Re: [SOLVED] memory problem

#18 2022-12-17 22:49:36

Re: [SOLVED] memory problem

#19 2022-12-18 00:23:45

Re: [SOLVED] memory problem

#20 2022-12-18 02:21:00

Re: [SOLVED] memory problem

#21 2022-12-18 02:24:48

Re: [SOLVED] memory problem

#22 2022-12-18 03:35:54

Re: [SOLVED] memory problem

#23 2022-12-18 06:15:14

Re: [SOLVED] memory problem

#24 2022-12-18 08:48:16

Re: [SOLVED] memory problem

#25 2022-12-18 11:49:07

Re: [SOLVED] memory problem

Board footer