KACE Product Support : Upgrading to 5.5 Problems

Upgrading to 5.5 Problems

We upgraded our appliance to 5.5 Friday and immediately started having issues. Using the System Admin Tools -> Top we can see our CPU usage on mysql running between 300% to 400%. Machines will not checking and system becomes unusable because of slowness.

I have opened a ticket with kace and installed 6 or 7 hotfixes with no help. They are asking use to rewrite about 300+ smart labels to see if this will help. These smart labels are nothing special and were all built using the “create smart label” option.

Anyone else having this after upgrading to 5.5?

0 Comments [ + ] Show comments

Answers (5)

Posted by: Gary.Cooper 11 years ago

We had the same issue after upgrading to 5.5. After working with support they had us basically do the same fixes that you listed above. We installed 7-8 patches and they went in and updated the 15 smart labels we have setup. In addition to these fixes, the one thing that was done on our support call that seemed to have corrected the problem was turning off classic metering. 5.5 has a new option for metering and the classic way was causing our agent tasks to become unresponsive. Here is the link that I was sent in order to setup metering in 5.5: https://www.kace.com/support/resources/kb/solutiondetail?sol=SOL114739

Hope this helps you resolve the issue.

Comments:

Thank you for you update We have about 75 classic metering jobs active. I will disable them and give that a shot.

I have modified the sql on about 80 machine smart labels so far. I still have about 300ish more to look at for machine and patching.

Just curious did you talk to a kace engineer for assistance? - young020 11 years ago
- I did not talk to a KACE engineer, it was first line support that assisted me. One thing I want to mention, once you disable the classic metering, restart the AMP Server from the Agent Messaging Protocol under settings. - Gary.Cooper 11 years ago

Posted by: ms01ak 11 years ago

Have you upgraded the clients as well to 5.5? I found that it major releases it can cause httpstorming and that causes the appliance to bog down. A simple fix would be put the appliance on a different ip address or to restart the amp server. Just some ideas.

Comments:

I would have to lean toward this as a known issue.

Make sure all of your KACE clients are up to date on 5.5. - Wildwolfay 11 years ago

Posted by: AbhayR 11 years ago

Can you run the command "database" from the System admin tools and see what mysql queries are running when mysql usage is 300% ?

Comments:

| 51569784 | B1 | localhost | ORG1 | Query | 10 | Sending data | select HD_TICKET.*,
HD_STATUS.NAME AS STATUS_NAME,
H |
|
| 51575231 | B1 | localhost | ORG1 | Query | 6 | Waiting for table level lock | select HD_TICKET.*,
HD_STATUS.NAME AS STATUS_NAME,
H |
|
| 51575510 | B1 | localhost | ORG1 | Query | 6 | Waiting for table level lock | select HD_TICKET.*,
HD_STATUS.NAME AS STATUS_NAME,
H |
|
| 51584654 | B1 | localhost | ORG1 | Query | 8 | Waiting for table level lock | UPDATE HD_TICKET SET TITLE='Network Infrastructure Repairs - Baseball Stadium',HD_PRIORITY_ID='1',HD |
| 51584655 | B1 | localhost | ORG1 | Query | 8 | Waiting for table level lock | select HD_TICKET.*, STATE
from HD_TICKET, HD_STATUS
|
| 51584658 | B1 | localhost | ORG1 | Query | 8 | Waiting for table level lock | select OWNER_ID, count(*)
from HD_TICKET, HD_STATUS
|
| 51584666 | B1 | localhost | ORG1 | Sleep | 6 | | NULL |
| 51584695 | B1 | localhost | ORG1 | Query | 6 | Waiting for table level lock | SELECT COUNT(*) FROM (HD_TICKET, HD_STATUS, HD_IMPACT, HD_CATEGORY)

It has a long list but i deleted out what showed nothing.

Looks like where the service desk is trying to update tickets. - young020 11 years ago
- Can you check from that list, which query is running for the longest time?
  
  It should be in the 6th column and most probably an INSERT/UPDATE/DELETE query - AbhayR 11 years ago
  - ADODB_EXTENSION: 5.04
    Warning: Using a password on the command line interface can be insecure.
    +----------+----------------------+--------------------+-------+---------+------+------------------------------+------------------------------------------------------------------------------------------------------+
    | Id | User | Host | db | Command | Time | State | Info |
    +----------+----------------------+--------------------+-------+---------+------+------------------------------+------------------------------------------------------------------------------------------------------+
    | 139615 | R1 | 55.555.55.55:62537 | ORG1 | Sleep | 112 | | NULL |
    | 8065921 | R1 | 55.555.55.55:53102 | ORG1 | Sleep | 85 | | NULL |
    | 35169717 | root | localhost | KBSYS | Sleep | 8 | | NULL |
    | 40409378 | R2 | 55.555.55.55:51223 | NULL | Sleep | 85 | | NULL |
    | 40409383 | R2 | 55.555.55.55:51224 | ORG2 | Sleep | 438 | | NULL |
    | 48392047 | B1 | localhost | ORG1 | Sleep | 7210 | | NULL |
    | 48392177 | B1 | localhost | ORG1 | Sleep | 7209 | | NULL |
    | 48392243 | B1 | localhost | ORG1 | Sleep | 7209 | | NULL |
    | 48392379 | B1 | localhost | ORG1 | Sleep | 7209 | | NULL |
    | 48392496 | B1 | localhost | ORG1 | Sleep | 7209 | | NULL |
    | 48392758 | B1 | localhost | ORG1 | Sleep | 7209 | | NULL |
    | 48393104 | B1 | localhost | ORG1 | Sleep | 7208 | | NULL |
    | 48393208 | B1 | localhost | ORG1 | Sleep | 7208 | | NULL |
    | 48393534 | B1 | localhost | ORG1 | Sleep | 7207 | | NULL |
    | 48393793 | B1 | localhost | ORG1 | Sleep | 7206 | | NULL |
    | 49063703 | R7 | 55.555.55.55:56863 | NULL | Sleep | 85 | | NULL |
    | 49063723 | R7 | 55.555.55.55:56864 | ORG7 | Sleep | 539 | | NULL |
    | 49168498 | root | localhost | KBSYS | Sleep | 5693 | | NULL |
    | 49899033 | root | localhost | KBSYS | Sleep | 4052 | | NULL |
    | 50117217 | root | localhost | KBSYS | Sleep | 2954 | | NULL |
    | 51873597 | root | localhost | KBSYS | Sleep | 24 | | NULL |
    | 51875543 | B1 | localhost | ORG1 | Sleep | 21 | | NULL |
    | 51875575 | root | localhost | KBSYS | Sleep | 22 | | NULL |
    | 51875612 | root | localhost | KBSYS | Sleep | 21 | | NULL
    
    This is pretty much what is looks like the first few entries. then it gets down to where I see the ticket rules. they have low times on them around 20 or so. I took the ip addresses out. - young020 11 years ago

Posted by: AbhayR 11 years ago

Few questions:

How many ticket rules are defined approximately?

How many tickets are there overall?

Also, are you seeing mysql to eat up 300% CPU always or intermittently?

Are you on a physical appliance or a virtual ?

Comments:

We are on a Virtual K1000 we have 7 Orgs
19 active ticket rules in Org 1, I would have to check the other orgs
Org 1 we have over 111k tickets in org 1, it is our main org. Archiving is turned on for tickets older than a year.

I have not noticed the CPU issues until 5.5 upgrade on Friday. - young020 11 years ago
- Ok. What kind of issues are you having exactly? slow ui on some pages? or something else?
  
  Also, when you run the "top" command, what are the load averages? - AbhayR 11 years ago
  - Out of 9K+ machines we have 1 machine checking in every hour or so.
    
    We are unable to patch, unable to deploy software, service desk hangs up when opening pages, saving tickets etc.
    
    last pid: 13358; load averages: 5.10, 4.88, 4.93 up 4+22:46:04 15:36:18
    201 processes: 8 running, 193 sleeping
    
    Mem: 4616M Active, 7799M Inact, 473M Wired, 705M Cache, 214M Buf, 2226M Free
    Swap: 512M Total, 3180K Used, 509M Free
    
    PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
    1228 mysql 75 51 0 3188M 3125M select 3 0:01 394.19% mysqld
    9090 www 1 -8 0 331M 44288K biowr 6 1:42 36.47% httpd
    12068 www 1 66 0 331M 44408K select 4 0:07 14.60% httpd
    3339 www 1 20 0 331M 45940K lockf 1 0:43 12.50% httpd
    13259 www 1 20 0 331M 44136K lockf 4 0:02 11.96% httpd
    11917 www 1 4 0 331M 44348K kqread 6 0:07 10.99% httpd
    7372 www 1 53 0 330M 44148K select 6 0:30 10.69% httpd
    13223 www 1 20 0 331M 44028K lockf 3 0:02 8.89% httpd
    11043 www 1 52 0 337M 49172K CPU6 6 0:10 6.88% httpd
    12946 www 1 59 0 331M 45156K select 1 0:03 6.15% httpd
    12896 www 1 60 0 331M 44220K select 1 0:02 6.15% httpd
    13272 www 1 58 0 330M 37944K select 7 0:00 2.29% httpd
    11146 www 1 -8 0 332M 44340K piperd 7 0:07 2.10% httpd
    79584 root 1 4 0 487M 472M kqread 7 263:55 1.86% kmsgr.7.0-amd64
    1587 root 1 8 0 157M 33560K nanslp 5 37:52 1.86% php
    12066 www 1 54 0 331M 45012K select 0 0:06 1.86% httpd
    12221 www 1 48 0 331M 45516K CPU5 5 0:05 1.56% httpd
    11916 www 1 -8 0 331M 45204K piperd 7 0:06 1.27% httpd - young020 11 years ago

Posted by: AbhayR 11 years ago

Can you check the "load average" from the "top" command?

Comments:

last pid: 13358; load averages: 5.10, 4.88, 4.93 up 4+22:46:04 15:36:18 201 processes: 8 running, 193 sleeping - See more at: http://www.itninja.com/question/upgrading-to-5-5-problems#sthash.mw4kAQzs.dpuf

I replied to other question also shows more info - young020 11 years ago
- Can you try disabling all the ticket rules for a few hours and see if that eases the mysql cpu ? - AbhayR 11 years ago

Answer this question

Posted by:

View more:

KACE Product Support Questions

question