The group could be targeted in the same way as other disadvantaged groups, he told The Independent.
Boys are now out-numbered at university by girls.
And the final figures for those going to university in the UK last autumn showed a bigger drop in applications from boys than girls.
Girls are more likely to apply to university than boys and more likely to get places at the most selective institutions.
Mr Willetts says there is a “shocking waste of talent” among some young people not going to university.
He told The Independent that the Office for Fair Access (the university access watchdog), “can look at a range of disadvantaged groups – social class and ethnicity, for instance – when it comes to access agreements, so I don’t see why they couldn’t look at white, working-class boys”.
Offa is charged with making sure universities in England set themselves targets to increase applications and take-up of places from disadvantaged groups.
‘Shocking waste of talent’
Speaking on the Today programme on BBC Radio Four, Mr Willetts said while it was up to universities to decide who to admit, they could be doing more to help teenagers from poorer backgrounds to make the grade, for example by inviting them back for repeated summer schools, to raise their chances of getting good A-levels.
“There are groups that are under-performing. There is a shocking waste of talent of some young people that could really benefit from university that aren’t going there,” he said.
The final figures for last autumn’s university intake in the UK show a fall in applications from men which was four times that among women.
Just 30% of male school-leavers applied to university for autumn 2012, compared to 40% of female school-leavers, according to Ucas.
Mr Willetts told The Independent this was “the culmination of a decades-old trend in our education system which seems to make it harder for boys and men to face down the obstacles in the way of learning… That is a challenge for all policymakers and parties.”
He added: “I do worry about what looks like increasing under-performance by young men.”
Mr Willetts told Today universities in England had been told to spend about a third of the money they gained through increased tuition fees on “reaching out and improving access ” and that this amounted to “hundreds of millions of pounds”.
“We want to see that used as effectively as possible,” he said.
The poor performance of many white working class boys in schools is something that has been highlighted in the past, particularly by England’s schools watchdog Ofsted.
Results of national tests known as Sats taken by 10 and 11-year-olds in England show that children on free school meals do less well than their classmates, and the pattern continues to GCSE level.
Last year just 66% of those known to be eligible for free school meals reached the expected level in English and maths, compared with 82% of all other pupils.
Just 60% of white British boys on free school meals reached that level, while 68% of black British boys did so.
Universities generally say that the under-representation of certain groups at university is mostly because they are not getting the qualifications needed while at school.
Universities do run summer schools and other programmes aimed at encouraging applications from disadvantaged groups, but some politicians and campaigners would like them to do more.
But any suggestion that universities should be made to admit teenagers with lower grades than others because of their background can be met by accusations of “social engineering”.
Universities say they do take applicants’ background and potential in to account when deciding on places.
Academics represented by the University and College Union say they agree that more needs to be done “to convince certain groups that university is for them” – but say poorer teenagers will be put off by increased tuition fees.
Sally Hunt, general secretary of the UCU, said: “We need our brightest people pursuing their dreams.
“However, a host of recent government policies have made university a far more expensive option and the drop off in student applications suggests the increased price is a factor.”
The government says no one should be put off going to university by finance because fees are not paid up-front and teenagers from poorer homes qualify for bursaries and loans to cover their living costs.
Figures out on Thursday from Ucas show a 6.3% fall in university applications from 18-year-olds in the UK compared with the same time last year. England and Wales show the biggest falls.