I think it's because the array created by Excel is not 0-based, but 1-based, you can convert it like this:

object[] ints = ((Array)value).ToArray<object>();

With this helper method extension:

public static T[] ToArray<T>(this Array array)
{
    if (array == null)
        return null;

    if (array.Rank != 1)
        throw new NotSupportedException();

    var newArray = new T[array.GetLength(0)];
    var lb = array.GetLowerBound(0);
    for (int i = 0; i < newArray.Length; i++)
    {
        newArray[i] = (T)array.GetValue(i + lb);
    }
    return newArray;
}

If you use .NET 4, there are chances that the return type of Evaluate will be defined as dynamic. So with this code:

dynamic value = xl.Evaluate("={1,2,3,4,5}"); // or var instead of dynamic

You will get a System.InvalidCastException: Unable to cast object of type 'System.Object[*]' to type 'System.Object[]' because the new .NET 4 interop layer will try automatically to create a standard array. So you need to do this instead (like you do in your code):

object value = xl.Evaluate("={1,2,3,4,5}");
object[] ints = ((Array)value).ToArray<object>();