Troubleshooting

This page contains various tips which can help you troubleshoot and debug code with interfaces with libcloud.

Debugging

Note

If you are sharing debug output on any public medium such as our IRC channel or an issue tracker using Pastebin, Github Gists or a similar service, make sure to remove your credentials and any other data you consider private from the output.

Libcloud has a special debug mode which when enabled, logs all the outgoing HTTP requests and all the incoming HTTP responses. Output also includes cURL commands which can be used to re-produce the requests.

When this mode is enabled and paramiko library is installed (used for deployment), paramiko library log level is set to DEBUG which helps with debugging the deployment related issues.

To make the debugging easier, Libcloud will also automatically decompress the response body (if compressed) before logging it.

To enable it, set LIBCLOUD_DEBUG environment variable and make it point to a file where the debug output should be saved.

If the API returns JSON or XML in the response body which is not human friendly, you can also set LIBCLOUD_DEBUG_PRETTY_PRINT_RESPONSE environment variable which will cause the JSON or XML to be beautified / formated so it’s easier for humans to read it. Keep in mind that this only works for non-chunked responses.

Example 1 - Logging output to standard error

Note

Some Linux distributions don’t allow /dev/{stderr,stdout} to be opened in append mode which means paramiko logger will return and error that it can’t log to that file. In such scenario, it’s recommended you specify actual file path and then you can use “tail -F” in another terminal window to view the output in real-time.

If you want the output to be logged to the standard error (on Linux) you can set it to /dev/stderr:

  1. LIBCLOUD_DEBUG=/dev/stderr python my_script.py

Example output:

  1. # -------- begin 4431824872 request ----------
  2. curl -i -X GET -H 'Host: s3.amazonaws.com' -H 'X-LC-Request-ID: 4431824872' -H 'Content-Length: 0' -H 'User-Agent: libcloud/0.6.0-beta1 (Amazon S3 (standard))' 'https://s3.amazonaws.com:443/?AWSAccessKeyId=foo&Signature=bar'
  3. # -------- begin 4431824872:4431825232 response ----------
  4. HTTP/1.1 200 OK
  5. X-Amz-Id-2: 1234
  6. Server: AmazonS3
  7. Transfer-Encoding: chunked
  8. X-Amz-Request-Id: FFFFFFFFFF
  9. Date: Tue, 01 Nov 2011 22:29:11 GMT
  10. Content-Type: application/xml
  11. 171
  12. <?xml version="1.0" encoding="UTF-8"?>
  13. <ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>sada8932dsa8d30i</ID><DisplayName>kami</DisplayName></Owner><Buckets><Bucket><Name>test34324323</Name><CreationDate>2011-11-01T22:17:23.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>
  14. 0
  15. # -------- end 4431824872:4431825232 response ----------

Example 2 - Making JSON / XML response human friendly

Non-formatted JSON response:

  1. LIBCLOUD_DEBUG=/dev/stderr python my_script.py
  1. # -------- begin 23125648:23160304 response ----------
  2. HTTP/1.1 200 OK
  3. Content-Length: 1572
  4. X-Compute-Request-Id: req-79ab42d8-a959-44eb-8dec-bc9458b2f4b3
  5. Server: nginx/1.4.7
  6. Connection: keep-alive
  7. Date: Sat, 06 Sep 2014 14:13:37 GMT
  8. Content-Type: application/json
  9. {"servers": [{"status": "ACTIVE", "updated": "2014-09-06T14:13:32Z", "hostId": "561d56de25c177c422278d7ca5f8b210118348040b12afbad06f278a", "addresses": {"internet-routable": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:3f:c0:a1", "version": 4, "addr": "10.100.100.101", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://nova/v2/d3b31ebfd32744d19d848f3e9c351869/servers/deb35f96-be41-431e-b931-6e615ec720f4", "rel": "self"}, {"href": "http://nova/d3b31ebfd32744d19d848f3e9c351869/servers/deb35f96-be41-431e-b931-6e615ec720f4", "rel": "bookmark"}], "key_name": null, "image": {"id": "e9537ddd-6579-4473-9898-d211ab90f6d3", "links": [{"href": "http://nova/d3b31ebfd32744d19d848f3e9c351869/images/e9537ddd-6579-4473-9898-d211ab90f6d3", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2014-09-06T14:13:32.000000", "flavor": {"id": "90c2a137-611b-4dd2-9d65-d4a0b0858531", "links": [{"href": "http://nova/d3b31ebfd32744d19d848f3e9c351869/flavors/90c2a137-611b-4dd2-9d65-d4a0b0858531", "rel": "bookmark"}]}, "id": "deb35f96-be41-431e-b931-6e615ec720f4", "security_groups": [{"name": "default"}], "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "06dda7c06aa246c88d7775d02bc119ac", "name": "test lc 2", "created": "2014-09-06T14:13:12Z", "tenant_id": "d3b31ebfd32744d19d848f3e9c351869", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "metadata": {}}]}
  10. # -------- end 23125648:23160304 response ----------

Human friendly formatted JSON response:

  1. LIBCLOUD_DEBUG=/dev/stderr LIBCLOUD_DEBUG_PRETTY_PRINT_RESPONSE=1 python my_script.py
  1. # -------- begin 41102928:41133624 response ----------
  2. HTTP/1.1 200 OK
  3. Content-Length: 1572
  4. X-Compute-Request-Id: req-3ce8b047-55cd-4e20-bfeb-b65619696aec
  5. Server: nginx/1.4.7
  6. Connection: keep-alive
  7. Date: Sat, 06 Sep 2014 14:14:38 GMT
  8. Content-Type: application/json
  9. {
  10. "servers": [
  11. {
  12. "OS-DCF:diskConfig": "MANUAL",
  13. "OS-EXT-AZ:availability_zone": "nova",
  14. "OS-EXT-STS:power_state": 1,
  15. "OS-EXT-STS:task_state": null,
  16. "OS-EXT-STS:vm_state": "active",
  17. "OS-SRV-USG:launched_at": "2014-09-06T14:13:32.000000",
  18. "OS-SRV-USG:terminated_at": null,
  19. "accessIPv4": "",
  20. "accessIPv6": "",
  21. "addresses": {
  22. "internet-routable": [
  23. {
  24. "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:3f:c0:a1",
  25. "OS-EXT-IPS:type": "fixed",
  26. "addr": "10.100.100.101",
  27. "version": 4
  28. }
  29. ]
  30. },
  31. "config_drive": "",
  32. "created": "2014-09-06T14:13:12Z",
  33. "flavor": {
  34. "id": "90c2a137-611b-4dd2-9d65-d4a0b0858531",
  35. "links": [
  36. {
  37. "href": "http://nova/d3b31ebfd32744d19d848f3e9c351869/flavors/90c2a137-611b-4dd2-9d65-d4a0b0858531",
  38. "rel": "bookmark"
  39. }
  40. ]
  41. },
  42. "hostId": "561d56de25c177c422278d7ca5f8b210118348040b12afbad06f278a",
  43. "id": "deb35f96-be41-431e-b931-6e615ec720f4",
  44. "image": {
  45. "id": "e9537ddd-6579-4473-9898-d211ab90f6d3",
  46. "links": [
  47. {
  48. "href": "http://nova/d3b31ebfd32744d19d848f3e9c351869/images/e9537ddd-6579-4473-9898-d211ab90f6d3",
  49. "rel": "bookmark"
  50. }
  51. ]
  52. },
  53. "key_name": null,
  54. "links": [
  55. {
  56. "href": "http://nova/v2/d3b31ebfd32744d19d848f3e9c351869/servers/deb35f96-be41-431e-b931-6e615ec720f4",
  57. "rel": "self"
  58. },
  59. {
  60. "href": "http://nova/d3b31ebfd32744d19d848f3e9c351869/servers/deb35f96-be41-431e-b931-6e615ec720f4",
  61. "rel": "bookmark"
  62. }
  63. ],
  64. "metadata": {},
  65. "name": "test lc 2",
  66. "os-extended-volumes:volumes_attached": [],
  67. "progress": 0,
  68. "security_groups": [
  69. {
  70. "name": "default"
  71. }
  72. ],
  73. "status": "ACTIVE",
  74. "tenant_id": "d3b31ebfd32744d19d848f3e9c351869",
  75. "updated": "2014-09-06T14:13:32Z",
  76. "user_id": "06dda7c06aa246c88d7775d02bc119ac"
  77. }
  78. ]
  79. }
  80. # -------- end 41102928:41133624 response ----------

Non-formatted XML response:

  1. LIBCLOUD_DEBUG=/dev/stderr python my_script.py
  1. # -------- begin 33145616:33126160 response ----------
  2. HTTP/1.1 200 OK
  3. X-Amzn-Requestid: e84f62d0-368e-11e4-820b-8bf013dc269e
  4. Date: Sun, 07 Sep 2014 13:00:13 GMT
  5. Content-Length: 457
  6. Content-Type: text/xml
  7. <?xml version="1.0"?>
  8. <ListHostedZonesResponse xmlns="https://route53.amazonaws.com/doc/2012-02-29/"><HostedZones><HostedZone><Id>/hostedzone/Z14L0C73CHH1DN</Id><Name>example1.com.</Name><CallerReference>41747982-568E-0DFC-8C11-71C23757C740</CallerReference><Config><Comment>test</Comment></Config><ResourceRecordSetCount>9</ResourceRecordSetCount></HostedZone></HostedZones><IsTruncated>false</IsTruncated><MaxItems>100</MaxItems></ListHostedZonesResponse>
  9. # -------- end 33145616:33126160 response ----------

Human friendly formatted XML response:

  1. LIBCLOUD_DEBUG=/dev/stderr LIBCLOUD_DEBUG_PRETTY_PRINT_RESPONSE=1 python my_script.py
  1. # -------- begin 19444496:19425040 response ----------
  2. HTTP/1.1 200 OK
  3. X-Amzn-Requestid: 01c02441-368f-11e4-b616-9b9bd7509a8f
  4. Date: Sun, 07 Sep 2014 13:00:56 GMT
  5. Content-Length: 457
  6. Content-Type: text/xml
  7. <?xml version="1.0" ?>
  8. <ListHostedZonesResponse xmlns="https://route53.amazonaws.com/doc/2012-02-29/">
  9. <HostedZones>
  10. <HostedZone>
  11. <Id>/hostedzone/Z14L0C73CHH1DN</Id>
  12. <Name>example1.com.</Name>
  13. <CallerReference>41747982-568E-0DFC-8C11-71C23757C740</CallerReference>
  14. <Config>
  15. <Comment>test</Comment>
  16. </Config>
  17. <ResourceRecordSetCount>9</ResourceRecordSetCount>
  18. </HostedZone>
  19. </HostedZones>
  20. <IsTruncated>false</IsTruncated>
  21. <MaxItems>100</MaxItems>
  22. </ListHostedZonesResponse>
  23. # -------- end 19444496:19425040 response ----------