Nitrogen (N) in the litter and soil inorganic N can have contrasting effects on litter decomposition, such that the positive effects of litter N on the rate of decomposition will likely decrease with increased soil inorganic N due to possible suppression of lignin oxidase. To test this, we reciprocally buried litter collected from multiple treatments in an N addition experiment (0, 20, 50, and 100 kg N ha−1 yr−1). With increasing N addition, we found that the concentration of N in the litter and the availability of inorganic N in the soil increased linearly. Contrary to our expectations, the litter N effects on decomposition changed nonlinearly with the N addition, first decreasing with the N addition rates but then reversed at the highest N addition treatment. This reversal at higher N rates is mainly due to a change in soil decomposing microbes with N addition. We examined the activity of extracellular enzymes in soil and found that phenol oxidase activity was not affected by N addition, but cellulase activity increased linearly with N addition rates. We further found that the relative abundance of bacterial functional genes involved in cellulose and the abundance of soft-rot fungi that decay cellulose increased with N addition. Our study contrasts the assumption that increased soil inorganic N suppresses lignin oxidation and instead reveals an increase in cellulolytic groups and activity of cellulase; this shift in decomposing microbes mediates the litter N effects on decomposition and suggests that more slowly decomposing fractions might be less affected.