Fixing “Object of type int64 is not JSON serializable” in Python

In this comprehensive guide, we’ll cover:

• What causes the “int64 not JSON serializable” error
• How to fix it by serializing NumPy ints and floats
• Alternate solutions like casting to Python types
• Using JSON encoders to serialize custom classes
• Best practices for handling JSON serialization

Follow along with code examples to learn the most effective ways to solve this common JSON serialization issue in Python.

Understanding the JSON Serialization Error

JSON is a text-based data format that can only represent certain data types:

• Strings
• Numbers (floats or integers)
• Booleans
• Lists
• Dicts
• Null

Trying to encode a Python object containing non-JSON compatible datatypes like int64 will result in a TypeError:

import json

data = {
  'x': 123,
  'y': np.int64(456) 
}

json.dumps(data)

# TypeError: Object of type int64 is not JSON serializable

This occurs because the NumPy int64 type does not have a direct JSON equivalent. We need to convert it to a supported type first before serializing it to JSON.

The same issue also applies to other non-JSON compatible Python types like sets, tuples, NaN, infinity, and Python-specific classes.

So in summary, the core reasons for the “int64 not JSON serializable” error are:

• Trying to serialize an incompatible data type like a NumPy int64
• Forgetting to convert incompatible Python objects like sets before serializing
• Attempting to encode custom class instances without a JSON encoder

Knowing what causes it points us to the solutions – converting those problem data types to JSON-safe values before attempting to dump into JSON.

Fix 1: Serialize NumPy ints and floats

A common source of “int64 not JSON serializable” errors is attempting to encode NumPy integer or float data types like np.int64, np.float32, etc.

Since JSON only supports standard floats and ints, we need to convert NumPy number types using:

• .item() to convert to standard Python int/float
• .asscalar() to get a Python int/float as well

import json
import numpy as np

data = {
  'x': 123,
  'y': np.int64(456)
} 

# Fix by converting np.int64 to Python int
data['y'] = data['y'].item() 

json.dumps(data) # No error!

# Or use .asscalar()
data['y'] = data['y'].asscalar()  
json.dumps(data) # Success

This resolves the issue by changing the NumPy int64 into a standard JSON-compatible Python int before serializing.

The same method applies for any NumPy number types like np.float32:

data = {
  'x': 123,
  'y': np.float32(45.6)
}

data['y'] = data['y'].item()
json.dumps(data) # Works!

So remember to convert NumPy values to native Python types when JSON encoding.

Fix 2: Cast to Python primitives

Another fix is to cast incompatible Python types like sets, tuples, and NaN/infinity values to primitive types like lists and floats that can be JSON encoded:

import json

data = {
  'x': 123,
  # Sets not JSON serializable 
  'y': {1, 2, 3}  
}

# Cast set to list
data['y'] = list(data['y']) 

json.dumps(data) # No more error!

We can also cast tuples to lists:

data = {
  'x': 123,
  'y': (1, 2, 3)
}

# Cast tuple to list
data['y'] = list(data['y'])  

json.dumps(data) # Success

And convert special float values:

import numpy as np

data = {
  'x': 123,
  'y': np.nan
}

# Convert NaN to None
data['y'] = None

json.dumps(data) # Works now!

This technique gives you full control to sanitize any unsupported Python types before attempting to serialize the data to JSON.

Fix 3: Use custom JSON encoders

If you need to serialize custom Python class instances to JSON, you can’t directly cast them to a primitive type. Instead, you need to create a custom encoder class to handle converting your custom objects.

Here is an example encoder to handle serializing a Person class instance:

from json import JSONEncoder

class Person:
  def __init__(self, name, age):
    self.name = name
    self.age = age

# Custom encoder
class PersonEncoder(JSONEncoder):       
  def default(self, o):
    if isinstance(o, Person):
      return {'name': o.name, 'age': o.age}
    return super().default(o)

person = Person('John', 30)

person_json = PersonEncoder().encode(person) 
print(person_json)
# '{"name": "John", "age": 30}'

The encoder’s default() method checks if the object is a Person instance and returns a dict of its attributes to serialize it properly.

You can pass your custom encoder to JSON. dumps() to use it:

json.dumps(person, cls=PersonEncoder)

This is a robust way to handle serializing classes, ndarray, datetime, and other custom Python objects you need to convert to JSON.

Best Practices for Fixing JSON Errors

Here are some best practices for avoiding and handling “Object of type XXX is not JSON serializable” errors:

• Check all your data types before encoding and sanitize as needed
• Convert incompatible types like NumPy ints/floats to native Python primitives
• Cast Python types like sets and tuples to JSON-friendly lists
• Handle NaN, infinity, etc. by converting to None or standard floats
• Create custom encoder classes to serialize custom Python classes
• Add error handling in case invalid data causes exceptions
• Use JSON schema validation to catch issues with types proactively
• Standardize on JSON-compatible data types like dicts when possible

Following these tips will help you smooth out the serialization process and produce valid JSON from your Python data.

Example Code for Fixing int64 Serialization

Here is some complete code showing different methods to fix the “Object of type int64 is not JSON serializable” error:

import json
import numpy as np

# Sample data
data = {
  'str': 'Text',
  'int': 123,
  'flt': 45.6,
  'tup': (1,2,3),
  'set': {1,2,3},
  'nan': np.nan,
  'arr': np.array([1,2,3]),
  'int64': np.int64(15)  
}

# Fix 1: Convert NumPy int64 to Python int
data['int64'] = data['int64'].item() 

print(json.dumps(data))

# Fix 2: Cast set and tuple to list
data['set'] = list(data['set'])
data['tup'] = list(data['tup'])

print(json.dumps(data))

# Fix 3: Convert NaN to None
data['nan'] = None

print(json.dumps(data)) 

# Fix 4: Custom encoder
class NumpyArrayEncoder(json.JSONEncoder):
  def default(self, obj):
    if isinstance(obj, np.ndarray):
      return obj.tolist()
    return json.JSONEncoder.default(self, obj)

print(json.dumps(data, cls=NumpyArrayEncoder))

This covers handling all the major cases you’re likely to encounter: NumPy ints, sets, tuples, NaN, and ndarray.

Combining these fixes provides robust protection against the notorious “Object of type XXX is not JSON serializable” errors in Python. You can confidently handle serializing complex data to JSON.

Conclusion

Dealing with “Object of type int64 is not JSON serializable” errors in Python simply comes down to:

• Understanding what Python types can’t be directly encoded to JSON
• Checking for and converting those types to JSON-compatible ones
• Using custom JSON encoders for custom class serialization

Following the examples in this guide, you should now have all the knowledge to smoothly handle converting Python objects to JSON.

Just remember to sanitize your data, use primitive types where possible, and implement encoders to serialize custom classes.